12 : international 
conference 


celda 2015 


’and, 


cognition-^ “"-•^thinking 

language exploratory 

32 W learning 

observation l ^ t ech/iology ^ 

-j. | * m focus 

digital- knowledge^*. 

age 


» 

- 4 — » 

03 

24 to 26 October 2015 
Maynooth, Greater Dublin, Ireland 


Proceedings 

Edited by: 

Demetrios G, Sampson, 
J. Michael Spector, 
Dirk Ifenthaler and 
Pedro Isaias 



kdfis 

international association for development of the information society 


12 th INTERNATIONAL CONFERENCE 


COGNITION AND 
EXPLORATORY LEARNING IN 
THE DIGITAL AGE 
(CELDA 2015) 




PROCEEDINGS OF THE 
12 th INTERNATIONAL CONFERENCE 

on 

COGNITION AND 
EXPLORATORY LEARNING IN 
THE DIGITAL AGE 
(CELDA 2015) 


MAYNOOTH, GREATER DUBLIN, IRELAND 
OCTOBER 24-26, 2015 


Organised by 

IADIS 

International Association for Development of the Information Society 


Co-Organised by 



Maynooth 

University 

National University 
of Ireland Maynooth 


Copyright 2015 
IADIS Press 
All rights reserved 

This work is subject to copyright. All rights are reserved, whether the whole or part of the 
material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, 
recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data 
banks. Permission for use must always be obtained from IADIS Press. Please contact 
secretariat@iadis.org 


Edited by Demetrios G. Sampson, J. Michael Spector, 
Dirk Ifenthaler and Pedro Isaias 

Associate Editor: Luis Rodrigues 


ISBN: 978-989-8533-43-2 



TABLE OF CONTENTS 


FOREWORD xi 

PROGRAM COMMITTEE xiii 

KEYNOTE LECTURE xv 

FULL PAPERS 

TOWARDS A METADATA SCHEMA FOR CHARACTERIZING LESSON PLANS 3 

SUPPORTED BY VIRTUAL AND REMOTE LABS IN SCHOOL SCIENCE 
EDUCATION 

Panagiotis Zervcis, Eleftheria Tsourlidaki, Sofoklis Sotiriou and Demetrios G. Sampson 

DIVULGING PERSONAL INFORMATION WITHIN LEARNING ANALYTICS 1 1 

SYSTEMS 

Dirk Ifenthaler and Clara Schumacher 

TAGALONG: INFORMAL LEARNING FROM A REMOTE COMPANION 19 

WITH MOBILE PERSPECTIVE SHARING 

Scott W. Greenwald, Mina Khan, Christian D. Vazquez and Pattie Maes 

ACCOUNTING PROFESSOR QUALIFICATION IN DIGITAL AGE: 28 

A PERCEPTION STUDY ON BRAZILIAN PROFESSORS 

Maria Ivanice Vendruscolo and Patricia Alejandra Behar 

SUPERVISORY AND DIGITAL LITERACY PRACTICES IN POSTGRADUATE 35 

SUPERVISION: A CASE STUDY 

Sarjit Kaur, Gurnam Kaur Sidhu, Lee Lai Fong and Leele Suzana Jamian 

POSTGRADUATE STUDENTS’ LEVEL OF DEPENDENCE ON SUPERVISORS 43 

IN COPING WITH ACADEMIC MATTERS AND USING DIGITAL TOOLS 

Gurnam Kaur Sidhu, Sarjit Kaur, Lim Peck Choo and Chan Yuen Fook 

TOWARDS GATHERING INITIAL REQUIREMENTS OF DEVELOPING 5 1 

A MOBILE SERVICE TO SUPPORT INFORMAL LEARNING AT CULTURAL 
HERITAGE SITES 

Alaa Alkhafaji, Sanaz Fallahkhair and Mihaela Cocea 

THE DEVELOPMENT AND EMPOWERMENT OF MATHEMATICAL ABILITIES: 59 

THE IMPACT OF PENCIL AND PAPER AND COMPUTERISED INTERVENTIONS 
FOR PRESCHOOL CHILDREN 

Maria Lidia Mascia, Mirian Agus, Maria Chiara Fastame, Maria Pietronilla Penna, 

Eliana Sale and Eliano Pessa 

THE RELATIONSHIP AMONG SELF-REGULATED LEARNING, 67 

PROCRASTINATION, AND LEARNING BEHAVIORS IN BLENDED LEARNING 
ENVIRONMENT 

Masanori Yamada, Yoshiko Goda, Takeshi Matsuda, Hiroshi Kato and Hiroyuki Miyagawa 


V 



75 


SOUND AS AFFECTIVE DESIGN FEATURE IN MULTIMEDIA LEARNING - 
BENEFITS AND DRAWBACKS FROM A COGNITIVE LOAD THEORY 
PERSPECTIVE 

Anke Konigschulte 

THE WILL, SKILL, TOOL MODEL OF TECHNOLOGY INTEGRATION: 84 

ADDING PEDAGOGY AS A NEW MODEL CONSTRUCT 

Gerald Knezek and Rhonda Christensen 

CHALLENGES OF BIG DATA IN EDUCATIONAL ASSESSMENT 92 

David C. Gibson, Mary ; Webb and Dirk Ifenthaler 

EVALUATING THE INTERACTIVE LEARNING TOOL SIMREAL+ 1 0 1 

FOR VISUALIZING AND SIMULATING MATHEMATICAL CONCEPTS 

Said Hadjerrouit 

PREDICTING THE RISK OF ATTRITION FOR UNDERGRADUATE STUDENTS 1 09 

WITH TIME BASED MODELLING 

Kevin EK Chai and David Gibson 

INSIDE THE DIGITAL WILD WEST: HOW SCHOOL LEADERS BOTH ACCESS 117 
AND AVOID SOCIAL MEDIA 

Laurie Corrigan and Lorayne Robertson 

MORE THAN A BROKER: A CASE STUDY OF KNOWLEDGE MOBILIZATION 125 
IN A DIGITAL ERA 

Lorayne Robertson 

HOW DO HIGH SCHOOL STUDENTS PREFER TO LEARN? 133 

Leila A. Mills and Putthachat Angnakoon 

PROCESS MODELS IN E-LEARNING - BOTTOM-UP OR TOP-DOWN? 140 

Sarah Sahl and Alke Martens 

A MODEL FOR UBIQUITOUS SERIOUS GAMES DEVELOPMENT FOCUSED 147 

ON PROBLEM BASED LEARNING 

Sandro Oliveira Dorneles, Cristiano Andre da Costa and Sandro Jose Rigo 

BLENDED INTERACTION FOR AUGMENTED LEARNING-AN ASSISTIVE 155 

TOOL FOR COGNITIVE DISABILITY 

Suman Deb, Subir Saha and Paritosh Bhattacharya 

STUDIES RELATING TO COMPUTER USE OF SPELLING AND GRAMMAR 163 

CHECKERS AND EDUCATIONAL ACHIEVEMENT 
Odette Bourjaili Radi 

DESIGN THINKING AND METACOGNITIVE REFLECTIVE SCAFFOLDS: 173 

A GRAPHIC DESIGN-INDUSTRIAL DESIGN TRANSFER CASE STUDY 

Chien-Sing Lee and Kuok-Shoong Daniel Wong 

DEVELOPING A DISPOSITION FOR SOCIAL INNOVATIONS: 180 

AN AFFECTIVE-SOCIO-COGNITIVE CO-DESIGN MODEL 

Chien-Sing Lee and Kuok-Shoong Daniel Wong 

TECHNOLOGY GOES BUSH: USING MOBILE TECHNOLOGIES TO SUPPORT 187 
LEARNING IN A BUSH KINDER PROGRAM 

Jennifer Masters and Leanne Grogan 

MOVING IN TIME TO A DIGITAL TUNE: A CRISIS IN OUR IDENTITY? 195 

Breda Me Taggart 


VI 



EXAMINING THE EFFECTS OF FIELD TRIPS ON SCIENCE IDENTITY 
Leila A. Mills and William Katzman 


202 


EDUCATION ON THE CLOUD: RESEARCHING STUDENT-CENTERED, 209 

CLOUD-BASED LEARNING PROSPECTS IN THE CONTEXT OF A EUROPEAN 
NETWORK 

Hercules Panoutsopoulos, Karl Donert, Panos Papoutsis and Ioannis Kotsanis 

UNIVERSITY AND FLIPPED LEARNING TIC&DIL PROJECT: FRAMEWORK 2 1 7 

AND DESIGN 

Stefania Pinnelli and Andrea Fiorucci 

TECHNOLOGY-ENHANCED PEDAGOGICAL FRAMEWORK FOR 225 

COLLABORATIVE CREATIVITY: ANALYSES OF STUDENTS’ PERCEPTION 

Manoli Pifarre, Laura Marti and Andreea Cujba 

COLLABORATIVE PROBLEM SOLVING IN SHARED SPACE 233 

Lin Lin, Leila A. Mills and Dirk Lfenthaler 

A SOCIAL NETWORKS IN EDUCATION 240 

Blanka Klimova and Petra Poulova 


SHORT PAPERS 


THE WORK OF CHILDREN: SEEKING PATTERNS IN THE DESIGN 249 

OF EDUCATIONAL TECHNOLOGY 

Michael Eisenberg and Zack Jacobson-Weaver 

TOWARDS SUPPORTING COMMUNICATION IN RELATIONSHIP 253 

AND SEXUALITY EDUCATION THROUGH A VLE 

Marion McGinn and Inmaculada Arnedillo-Sdnchez 

THE DEVELOPMENT OF COMPUTATIONAL THINKING IN THE CONTEXT 257 

OF SCIENCE AND ENGINEERING PRACTICES: A SELF-REGULATED 
LEARNING APPROACH 

Erin E. Peters-Burton, Timothy J. Cleary > and Anastasia Kitsantas 

A PROGRAM COMPLEXITY METRIC BASED ON VARIABLE USAGE 262 

FOR ALGORITHMIC THINKING EDUCATION OF NOVICE LEARNERS 

Minori Fuwa, Mizue Kayama, Hisayoshi Kunimune, Masami Hashimoto and David K Asano 

APPS. ACCESSIBILITY AND USABILITY BY PEOPLE WITH VISUAL 267 

DISABILITIES 

Eva Maria Olmedo-Moreno and Adrian Lopez-Delgado 

COMPARING LEARNER COMMUNITY BEHAVIOR IN MULTIPLE 271 

PRESENTATIONS OF A MASSIVE OPEN ONLINE COURSE 

Silvia Elena Gallagher and Timothy Savage 

DEVELOPING ADOLESCENTS’ RESISTANCE TO SEXUAL COERCION 275 

THROUGH ROLE-PLAYING ACTIVITIES IN A VIRTUAL WORLD 

Marion McGinn and Inmaculada Arnedillo-Sdnchez 

HOW CAN ONE LEARN MATHEMATICAL WORD PROBLEMS IN A SECOND 279 
LANGUAGE? A COGNITIVE LOAD PERSPECTIVE 

Jase Moussa-Lnaty, Mark Causapin and Timothy Groombridge 


vii 



ONTOLOGICAL RELATIONS AND THE CAPABILITY MATURITY MODEL 
APPLIED IN ACADEMIA 

Jeronimo Moreira de Oliveira, Laura Gomez Campoy and Lilian Vilarino 


283 


COGNITIVE CURATIONS OF COLLABORATIVE CURRICULA 288 

Amy S. Ackerman 

HOW OLDER ADULTS USE VIRTUAL PERSONAL LEARNING NETWORKS 293 

TO SUPPORT INFORMAL, SELF-DIRECTED LIFELONG LEARNING GOALS: 

A RESEARCH PROGRAM DESCRIPTION 

Dirk Morrison 

SIGNIFICANT CHANGES IN THE ENVIRONMENT AND IN TEACHING 297 

METHODOLOGY OF A E-LEARNING DISCIPLINE TO AVOID DROPOUTS 
IN A COURSE AT THE FEDERAL INSTITUTE 

Gustavo Prado Oliveira, Pdivi Aarreniemi-Jokipelto and Ricardo Soares Boaventura 

E-FUNDI ASA VIABLE WAY TO DO E-MENTORING 3 0 1 

George A. Lotter 

HEALTH CARE: ROLE OF ICT IN AUTISM 305 

Hafiza Maimoona Shahid, Sohaib Tariq, Imran Saleem, Muhammad Adil Butt, Arslan Tariq and 
Iqra Tariq 

INVESTIGATION INTO UNDERGRADUATE INTERNATIONAL STUDENTS ’ 309 

USE OF DIGITAL TECHNOLOGY AND THEIR APPLICATION IN FORMAL 
AND INFORMAL SETTINGS 

Rebecca Strachan and Sanaa Aljabali 

E-LEARNING SYSTEM FOR LEARNING VIRTUAL CIRCUIT MAKING WITH 3 1 3 

A MICROCONTROLLER AND PROGRAMMING TO CONTROL A ROBOT 

Atsushi Takemura 

A STUDY ON THE EFFECTS OF THE RUBRIC ON CONCURRENT DISCUSSION 319 
IN WEB-BASED ENVIRONMENT 

Jaewon Jung 

BRAIN 3M - A NEW APPROACH TO LEARNING ABOUT BRAIN, BEHAVIOR, 323 
AND COGNITION 

Ping Li, Lauren E. Chaby, Jennifer Legault and Victoria A. Braithwaite 

LEARNER-CONTENT-INTERFACE AS AN APPROACH FOR SELF-RELIANT 327 

AND STUDENT-CENTERED LEARNING 

Robin Nicolay, Bastion Schwennigcke, Sarah Sahl and Alke Martens 

CACHE-CACHE COMPARISON FOR SUPPORTING MEANINGFUL LEARNING 332 

Jingyun Wang and Seiji Fujino 

CREATING INNOVATIVE, STUDENT-CENTERED PROJECTS WITH APP 337 

SMASHING 

Aimee M. Brenner and Jan el S. Hauser 

SMART LEARNING: ARE WE READY FOR IT? 341 

Petra Poulova and Blanka Klimova 

A FUNDAMENTAL STUDY FOR EFFICIENT IMPLEMENTAION OF ONLINE 347 

COLLABORATIVE ACTIVITIES IN LARGE-SCALE CLASSES 

Ryuichi Matsuba, Yusei Suzuki, Shin-Ichiro Kubota and Makoto Miyazaki 

ENCOURAGING USER PARTICIPATION IN BLENDED LEARNING: 351 

COURSE REORIENTATION 

Alea M. Fairchild 


viii 



ASSESSING PROBLEM SOLVING COMPETENCE THROUGH INQUIRY-BASED 355 
TEACHING IN SCHOOL SCIENCE EDUCATION 

Panagiotis Zervas, Sofoklis Sotiriou, Rudiger Tiemann and Demetrios G. Sampson 

MUSIC AS ACTIVE INFORMATION RESOURCE FOR PLAYERS IN VIDEO 359 

GAMES 

Marian Nagorsnick and Alke Martens 

MAKING CONSTRUALS AS A NEW DIGITAL SKILL FOR LEARNING 363 

Meurig Beynon, Russell Boyatt, Jonathan Foss, Chris Hall, Elizabeth Hudnott, Steve Russ, 

Erkki Sutinen, Hamish Macleod and Piet Kommers 

A SEMANTIC BASIS FOR MEANING CONSTRUCTION IN CONSTRUCTIVIST 369 
INTERACTIONS 

Farshad Badie 


REFLECTION PAPERS 


EXPLORING TECHNOLOGY SUPPORTED COLLABORATIVE 377 

AND COOPERATIVE GROUP FORMATION MECHANISMS 

Mia Carapina and Ivica Boticki 

DEVELOPING SELF-REGULATED LEARNERS THROUGH COLLABORATIVE 379 

ONLINE CASE DISCUSSION IN EDUCATIONAL PSYCHOLOGY 

Patricia P. Willems and Alyssa Gonzalez-DeHass 

STUDENT’S PERSPECTIVES ON TAKING COURSES ONLINE, BLENDED, 381 

OR A COMBINATION 

Catherine C. Schifter, Dirk Ifenthaler and Daniel White 

AN INNOVATIVE INTERDISCIPLINARY APPROACH TO PROVIDING 383 

INTERNSHIPS FOR COLLEGE SENIORS 

Kathy Fuller 

FOSTERING 21ST CENTURY SKILLS THROUGH GAME DESIGN 385 

AND DEVELOPMENT 

Gregory’ P. Garvey 

DEVELOPMENT OF CRITICAL THINKING SELF-ASSESSMENT SYSTEM 387 

USING WEARABLE DEVICE 

Yasushi Got oh 


AUTHOR INDEX 


IX 




FOREWORD 


These proceedings contain the papers of the 12 th International Conference on Cognition 
and Exploratory Learning in the Digital Age (CELDA 2015), 24-26 October 2015, 
which has been organized by the International Association for Development of the 
Information Society (IADIS), co-organized by Maynooth University, Ireland, and 
endorsed by the Japanese Society for Information and Systems in Education (JSISE). 

The CELDA 2015 conference aims to address the main issues concerned with evolving 
learning processes and supporting pedagogies and applications in the digital age. There 
have been advances in both cognitive psychology and computing that have affected the 
educational arena. The convergence of these two disciplines is increasing at a fast pace 
and affecting academia and professional practice in many ways. 

Paradigms such as just-in-time learning, constructivism, student-centered learning and 
collaborative approaches have emerged and are being supported by technological 
advancements such as simulations, virtual reality and multi-agents systems. These 
developments have created both opportunities and areas of serious concerns. This 
conference aims to cover both technological as well as pedagogical issues related to 
these developments. Main tracks have been identified. However innovative 
contributions that do not easily fit into these areas will also be considered as long as 
they are directly related to the overall theme of the conference - cognition and 
exploratory learning in the digital age. 


The following areas are represented in the submissions for CELDA 2015: 


• Acquisition of expertise 

• Assessing progress of learning in 
complex domains 

• Assessment of exploratory learning 
approaches 

• Assessment of exploratory 
technologies 

• Cognition in education 

• Collaborative learning 

• Educational psychology 

• Exploratory technologies 
(simulations, VR, i-TV, etc.) 

• Just-in-time and 
Leaming-on-Demand 


• Learner communities and 
peer-support 

• Learning communities & Web service 
technologies Pedagogical issues 
related with learning objects 

• Learning paradigms in academia 

• Learning paradigms in the corporate 
sector 

• Life-long learning 

• Student-centered learning 

• Technology and mental models 

• Technology 

• Learning and expertise 

• Virtual university 


The CELDA 2015 Conference received 92 submissions from more than 23 countries. 
Each submission was reviewed in a double -blind review process by at least two 
independent reviewers to ensure quality and maintain high standards. Out of the papers 
submitted, 31 were accepted as full papers for an acceptance rate of 34%; 28 were 
accepted as short papers and 6 were accepted as reflection papers. Authors of the best 


xi 



published papers in the CELDA 2015 proceedings will be invited to publish extended 
versions of their papers in a book from Springer. 

In addition to the presentation of full papers, short papers and reflection papers, the 
conference also includes a keynote presentation from an internationally distinguished 
researcher. We would therefore like to express our gratitude to Professor Mary J. 
Bishop, Director, Center for Academic Innovation, University System of Maryland, 
USA, as the CELDA 2015 keynote speaker. 

A successful conference requires the effort of many individuals. We would like to thank 
the members of the Program Committee for their hard work in reviewing and selecting 
the papers that appear in this book. We are especially grateful to the authors who 
submitted their papers to this conference and to the presenters who provided the 
substance of the meeting. We wish to thank ah members of our organizing committee. 

Last but not least, we hope that participants enjoy Maynooth and their time with 
colleagues from ah over the world. 

Pedro Isaias, Universidade Aberta (Portuguese Open University), Portugal 
Conference Chair 

Demetrios G Sampson, Curtin University, Australia 
J. Michael Spector, University of North Texas, USA 

Dirk Ifenthaler, University of Mannheim, Germany and Deakin University, Australia 
Program Co-Chairs 

Rose Dolan, Maynooth University, Ireland 
Local Organizing Chair 


Maynooth, Greater Dublin, Ireland 
October 2015 
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KEYNOTE LECTURE 


INSTRUCTIONAL MESSAGE DESIGN: PAST, PRESENT, AND 

FUTURE RELEVANCE 


Professor Mary J Bishop, 
Director, Center for Academic Innovation, 
University System of Maryland, USA 


Abstract 

Instructional message design explores how various media and delivery systems might 
be used more effectively to help optimize instructional communications within 
context-specific instructional situations and learner needs. But use of the tenn appears 
to have fallen out of favor over the years since the mid-1990s. A review of the historical 
and theoretical foundations of instructional message design reveals that, while 
instructional design generally has shifted from objectivist to more constructivist 
perspectives on learning theory, the instructional message design field remains firmly 
rooted in early “transmission oriented” communications models. It appears that 
instructional message design has also suffered from definitional problems as well, with 
more recent narrow views of the field focused on media attributes supplanting earlier 
broad views of the field as an applied “linking science” between theory and practice. 
And, finally, while findings from studies on media attributes provide designers with 
some guidance for generally what will not work in tenns of cognitive processing, the 
guidelines seldom shed light on what one should actually do within a particular learning 
context. Re-establishing instructional message design as a valid area of inquiry within 
the field of instructional design will require catching up with recent philosophical shifts 
in communication theory while adjusting our definitions and research foci accordingly. 

In this presentation I will briefly revisit instructional message design’s objectivist 
learning theory and transmission-oriented communication theory foundations, discuss 
current trends and issues being raised in the literature, and explore ways the field might 
continue to serve as a “linking science” between learning theory and instructional 
practice. I will conclude with recommendations for a revised guiding theoretical 
framework based in conversation theory, a broader definitional focus that looks at more 
than just optimizing cognitive processing, and a new systems view of our approach to 
research in this area. 
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TOWARDS A METADATA SCHEMA FOR 
CHARACTERIZING LESSON PLANS SUPPORTED BY 
VIRTUAL AND REMOTE LABS IN SCHOOL SCIENCE 

EDUCATION 


Panagiotis Zervas 1 , Eleftheria Tsourlidaki 2 , Sofoklis Sotiriou 2 and Demetrios G. Sampson 1 ' 3 

1 Information Technologies Institute (ITI), Centre for Research and Technology Hellas (CERTH) 

6th km Charilaou-Thermi Rd„ GR 57001, Thessaloniki, Greece 
2 Ellinogermaniki Agogi, Dimitriou Panagea Str. GR 153 51 Pallini Attikis, Athens, Greece 
3 School of Education, Curtin University, Kent Street, Bentley WA 6102, Australia 


ABSTRACT 

Technological advancements in the field of World Wide Web have led to a plethora of remote and virtual labs (RVLs) 
that are currently available online and they are offered with or without cost. However, using a RVL to teach a specific 
science subject might not be a straightforward task for a science teacher. As a result, science teachers need to be able to 
find existing lesson plans supported by RVLs (designed by other science teachers), so as to (a) be informed on how 
specific RVLs can be used in the context of a science education lesson and (b) be inspired and possibly adapt existing 
lesson plans supported by RVLs to cover their specific teaching needs. The most common way to facilitate this process 
using web technologies is to (a) characterize lesson plans with appropriately selected educational metadata and (b) to 
build a web repository that collects the metadata descriptions of lesson plans following a common metadata schema and 
offers search and retrieval facilities. Within this context, the scope of this paper is twofold: (a) to propose a metadata 
schema that can be used for characterizing school science education lesson plans supported by RVLs and (b) to validate 
this metadata schema with 82 European school science teachers that was performed in the framework of a major 
European Initiative namely, the Go-Lab project, so as to identify which metadata elements are considered important when 
science teachers are searching in web-based repositories. 


KEYWORDS 

School education, science education, inquiry-based learning, remote lab, virtual lab. web-based repository, metadata 
schema, lesson plan, validation study 


1. INTRODUCTION 

Remote and Virtual Labs (RVLs) constitute significant educational tools for supporting science teachers in 
their daily teaching practice, especially when adopting the inquiry-based teaching model (de Jong et al., 
2013). In particular, remote labs provide teachers with the opportunity to engage their students in the process 
of data collection data from a real physical laboratory, including real equipment from remote locations 
(Gomes & Bogosyan, 2009), whereas virtual labs constitute interactive environments for designing and 
conducting simulated experiments (Balamuralithara, & Woods, 2009). However, using a RVL to teach a 
specific science subject might not be a straightforward task for a science teacher, especially a novice one 
(Govaerts et al., 2013). As a result, science teachers could benefit from having access to existing lesson plans 
supported by RVLs (developed by other science teachers), so as to (a) be informed on how specific RVLs 
can be used in the context of a science inquiry-based education lesson and (b) be inspired and possibly adapt 
existing lesson plans supported by RVLs to cover their specific teaching needs. A lesson plan describes how 
a lesson should take place, which planned activities will be executed by students (individually or at groups) 
and teachers/tutors, the order in which the activities are planned to be executed, the required contextual 
conditions within which the activities will be executed, how learners will be grouped (if appropriate) and 
which educational resources and/or tools will be used for each activity (Van Es & Koper, 2005). 
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A key problem then emerges on how to support school science teachers to select and find lesson plans 
supported by RVLs in an efficient (that is, easily via simple web-based searches) and effective (that is, 
according to their teaching needs and conditions) way. The most common way to address this need by using 
web technologies is to: 

■ Characterize lesson plans with appropriately selected educational metadata (Dagiene & 
Kubilinskiene, 2010; Battigelli & Sugliano, 2009) 

■ Build a web application that facilitate lesson plans’ search and retrieval according to a specific user 
(that is, science teacher) defined requests, i.e. searching with specific elements of lessons plans such 
as subject domain, educational objectives or age range addressed. Such applications are typically 
build in the form of a web repository that collects the metadata descriptions of lesson plans 
following a common metadata schema and offers search and retrieval facilities (Dong et al., 2009) 

Within this context, the scope of this paper is twofold: (a) to propose a metadata schema that can be used 
for characterizing lesson plans supported by RVLs and (b) to validate this metadata schema with 82 
European school science teachers that was performed in the framework of a major European Initiative 
namely, the Go-Lab project 1 , so as to identify which metadata elements are considered important when 
science teachers are searching for lesson plans supported by RVLs in web-based repositories. 

The remainder of the paper is structured as follows. Following this introduction, section 2 reviews existing 
repositories of school education science lesson plans supported by RVLs and performs an analysis of the 
metadata elements used by these repositories towards identifying common metadata elements. Section 3 
presents the proposed metadata schema for characterizing RVL-supported school science education lesson 
plans, which consists of the synthesis of the metadata elements identified in section 2. Section 4, presents the 
methodology that was followed for validating the proposed metadata schema. Section 5 presents and discusses 
the validation results. Finally, we discuss our main conclusions and our future work in this agenda 


2. REVIEW OF REPOSITORIES OF LESSON PLANS SUPPORTED BY 
RVLS 

The aim of this section is to review existing repositories of school science education lesson plans supported 
by RVLs and identify common metadata elements used for characterizing these lesson plans. A set of ten 
( 10) repositories is reviewed. These repositories have been selected because their purpose is twofold: (a) they 
store and provide searching facilities for RVLs and (b) they store and provide searching facilities for lesson 
plans supported by these RVLs. As a result, we consider them as the most appropriate for review towards 
defining our proposed metadata schema. Table 1 presents the existing repositories that were reviewed, the 
number of lesson plans that they store (at the time of our study), as well as the number of metadata elements 
used by each repository for characterizing its lesson plans. 


Table 1. Overview of Existing Repositories of Lesson Plans supported by RVLs 


No 

Name 

Repository URL 

# Lesson 
Plans 2 

# Metadata Elements 
used 

1 

PhET 

http://phet.colorado.edu 

552 

13 

2 

Lab share 

http://www.labshare.edu.au/ 

12 

13 

3 

Explore Learning 

http://www. explorelearning .com 

478 

13 

4 

Open Sources Physics 

http://www.compadre.org/osp 

355 

12 

5 

Lab2Go 

http://www.lab2go.net 

N/A 

11 

6 

ChemCollective 

http://www. chemcollecti ve. or g/ 

55 

10 

7 

Intel Education Resources - 
STEM 

http://inteleducationresources.intel.co.uk/index.aspx 

264 

10 

8 

iLabCentral 

http://ilabcentral.org 

21 

10 

9 

Molecular Workbench 

http://mw.concord.org/ 

75 

3 

10 

Remotely Controlled 
Laboratories (RCL) 

http://rcl-munich.informatik.unibw-muenchen.de 

17 

3 


1 http://www.go-lab-project.eu/ 

2 Data retrieved on 22/5/2015 
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As Table 1 depicts, the number of metadata elements used by the different repositories to characterize the 
RVL-supported school science education lesson plans that they store, varies from a small number of elements 
(namely, 3 elements) to a larger number of elements (namely, 13 elements). This is due to the fact that each 
repository uses its own customized metadata schema for characterizing the RVL-supported school science 
education lesson plans that it stores without following a specific metadata standard (such as IEEE LOM 
(IEEE LTSC, 2005)). As a result, some of the metadata schemas adopted by the examined repositories are 
richer than others. Moreover, in some cases a metadata element used by the metadata schema adopted by one 
repository might not be used by others. To this end. in order to come up with an initial proposal about a 
metadata schema of lesson plans supported by RVLs, we harmonized the lesson plan’s metadata elements 
used by the repositories presented in Table 1 and we produced a master list (comprising a synthesis of the 
different metadata elements used by the examined repositories), which is presented in the next section. 


3. PROPOSED METADATA SCHEMA OF LESSON PLANS SUPPORTED 
BY RVLS 

As already mentioned, the starting point for developing our proposed initial metadata schema was the 
outcomes of the study of the metadata schemas of existing repositories of RVL-supported science education 
lesson plans presented in section 2. Table 2 presents our proposed initial metadata schema which consists of 
the synthesis of metadata elements identified from the review presented in section 2. Moreover, Table 2 
presents for each metadata element the frequency of use at the repositories that were reviewed in section 2. 


Table 2. Proposed Metadata Schema of Lesson Plans supported by RVLs 


No 

Element Name 

Description 

Taxonomy 

Available? 

Usage 

Frequency 

1 

Title 

This metadata element refers to the title of the lesson plan 

No 

10(100%) 

2 

URL 

This metadata element provides a URL for accessing the lesson plan 

No 

10(100%) 

3 

Description 

This metadata element provides a textual description of the lesson plan 

No 

8 (80%) 

4 

Subject Domain 

This metadata element refers to the lesson plan’s subject domain 

Yes 

8 (80%) 

5 

Lab(s) Used 

This metadata element denotes the online labs used in the lesson plan 

Yes 

8 (80%) 

f. 

Owner(s) and 

This metadata element provides information about the owner, as well as 

No 

8 (80%) 

0 

Contributor(s) 

Additional 
Materials included 

entities that have contributed to the authoring of the lesson plan 
This metadata element describes additional supportive material that can 



7 

facilitate teachers to deliver the lesson (based on the lesson plan) and 
students to execute the lesson 

No 

8 (80%) 

8 

Language(s) 

This metadata element refers to the languages that the lesson plan is 
available in. 

Yes 

7 (70%) 

9 

Age Range 

This metadata element refers to the age range for which the lesson plan 
can be used. 

Yes 

7 (70%) 

10 

Keyword(s) 

This metadata element refers to a set of terms that characterize the content 
of the lesson plan 

No 

5 (50%) 

11 

Status 

This metadata element provides information about the current status of the 
lesson plan. 

Yes 

4 (40%) 

12 

Educational 

Objectives 

This metadata element refers to the educational objectives that the lesson 
plan addresses 

Yes 

3 (30%) 

13 

Organizational 

This metadata element refers to the requirements that are needed in order 

No 

3 (30%) 

Requirements 

to carry out the lesson plan without troubleshooting. 

14 

Level of Difficulty 

This metadata element refers to the level of difficulty of the lesson plan. 

Yes 

2 (20%) 

15 

Average Learning 
Time 

This metadata element refers to the amount of time that the lesson plan 
requires in order to be completed 

Yes 

2 (20%) 

16 

Group Learning 

This metadata element indicates whether the lesson plan follows a specific 

Yes 

2 (20%) 

Method used 

group learning method (such as jigsaw, changing hats etc) 

18 

Access Rights 

This metadata element refers to the lesson plan’s access permissions 
This metadata element refers to the level of interaction the lesson plan 

Yes 

1 (10%) 

18 

Level of Interaction 

offers in terms of (a) variables manipulation during experimentation and 
(b) interaction and collaboration with peers 

Yes 

1 (10%) 

19 

Students’ prior 
Knowledge 

This metadata element refers to students’ prior knowledge in order to 
execute the lesson 

Yes 

1 (10%) 
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As we can notice from Table 2, our proposed initial metadata schema includes 19 metadata elements. 10 
out of 19 metadata elements are used in more than fifty percent (50%) of the repositories that were reviewed 
in section 2, whereas 9 out of 19 metadata elements are used in less than fifty percent (50%) of the reviewed 
repositories. The proposed metadata schema has been used for storing and classifying lesson plans in a 
web-based repository, namely the Go-Lab Repository (http://www.golabz.eu/) developed in the framework of 
the Go-Lab Project and presented in (Dikke et al., 2014). The Go-Lab project (http://www.go-lab-project.eu/) 
aims to establish an online portal, namely the Go-Lab Portal that will facilitate the federation of existing 
virtual and remote labs (de Jong et al., 2014; Govaerts et al., 2013). In the context of the Go-Lab project, a 
lesson plan comprises (a) a set of technology-supported activities structured in inquiry phases that follow 
parts of the full inquiry-based teaching model and they are executed individually by the students with the 
support of the Go-Lab Portal and the RVLs that includes (this set of activities is referred to as an inquiry 
learning space-ILS) and (b) a set of supplemental activities to the aforementioned technology-supported 
activities, which are not supported by technology and they are executed by the teacher or the students 
(individually or at groups) (de Jong et al., 2014). The Go-Lab Repository is part of the Go-Lab portal, it has 
been populated with 137 3 RVL-supported science education lesson plans, and provides functionalities for 
facilitating search and retrieval of these lesson plans by school science teachers. 


4. VALIDATING THE PROPOSED METADATA SCHEMA OF LESSON 
PLANS SUPPORTED BY RVLS 

4.1 Related Studies 

In order to develop our methodology for validating our proposed metadata schema, we reviewed existing 
works on validating metadata schemas with real users Since there are not related works that focus on 
validating metadata schemas for lesson plans supported by RVLs, the review is performed on related works 
for validating metadata schemas for educational resources in general. Table 3, presents briefly these studies 
along with their basic parameters. 


Table 3. Studies on Validating Metadata Schemas of Educational Resources 


Study 

Application Domain 

Validation Instrument 

Users 

Palavitsinis et al. (2009) 
Krull et al. (2006) 

Howarth (2003 ) 

Agriculture and Agroecology 
Interdisciplinary 

Interdisciplinary 

Questionnaire 

Questionnaire 

Questionnaire 

Subject Domain Experts 
Teachers/Trainers 
Subject Domain Experts and 
Teachers/Trainers 

Carey et al. (2002) 

Interdisciplinary 

Questionnaire & 
Interview 

Subject Domain Experts 


As we can notice from Table 3, commonly used validation instruments are questionnaires through which 
end-users (teachers/trainers or subject domain experts) are asked to validate one by one the various metadata 
elements of the proposed metadata schema. As a result, a similar methodology has been followed in our case 
and it is described in details in the next section. 

4.2 Study Methodology 

4.2.1 Sample 

The study was conducted with N=82 European School science teachers who were invited to be part of the 
Go-Lab Project pilot activities. Our sample included school science teachers from thirteen (13) European 
member states. Moreover, in our sample there was a gender balance between the participants (52% female, 
48% male). The majority of the participants were experienced science teachers (69% of the participants had 
more than 6 years of teaching experience). Furthermore, almost all science education school teachers in our 


3 Data retrieved on 13/7/2015 
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sample were experienced in using ICT for their daily teaching activities and the majority of them had 
previous experience in working with RVLs (66% using virtual labs and 23% using remote labs). Finally, 61% 
of them have at least a master's degree. Thus, overall we consider that our sample is well selected for the 
purpose of our study. 

4.2.2 Procedure 

In order to validate the proposed metadata schema, we provided to the participated science teachers a 
questionnaire. The aim of the questionnaire was to collect participants’ opinions on the importance of certain 
metadata elements from our proposed metadata schema. More precisely, the teachers were asked to rate each 
metadata element with a five -point like scale, where 1 denotes “low importance” and 5 denotes “high 
importance” for the following there different contexts of use: (a) importance of metadata elements within the 
context of “ making a general search for lesson plans ” in the Go-Lab repository, (b) importance of metadata 
elements within the context of ‘ filtering search results for lesson plans'' in the Go-Lab repository and (c) 
importance of metadata elements within the context of “viewing the preview page of a Go -Lab lesson plan" 
in the Go-Lab repository 

It should be noted that out of the 19 elements that are part of the metadata model, 17 were included in the 
questionnaire. Elements such as “Location URL” and “Description” were not included in the questionnaire, 
because we consider them essential for our proposed metadata model and thus no further investigation was 
needed on validating their importance. In order to receive feedback from the participants based on the 
questionnaire that was designed, appropriate workshops were organized that had the following structure: 

■ During the workshops: all participants attended a demonstration of the Go-Lab repository. More 
specifically, the main functionalities of the repository, as well as the search and retrieval facilities 
were presented, along with the lesson plans that were stored in the repository. Moreover, all 
participants could navigate within the Go-Lab repository during the workshop, through their 
personal devices (laptops or tablets) and use it themselves directly. Thus, they had a concrete idea of 
how lesson plan’s metadata elements have been deployed within the Go-Lab repository. 

■ After the workshops: all participants were asked to rate the lesson plan’s metadata elements by 
completing the designed questionnaire. 


5. RESULTS 

This section presents quantitative data analysis results for participants’ feedback regarding the importance of 
lesson plan’s metadata elements (see Table 4) for the different contexts of use (as described in section 4.2.1 ). 


Table 4. Ranking of the Importance of Metadata Elements 



Making a General Search 


Filtering Search Results 


Viewing the Preview Page 

Ranking 

Metadata Element 

Mean 

SD 

Metadata Element 

Mean 

SD 

Metadata 

Element 

Mean 

SD 

1 

Title 

4,39 

0,84 

Title 

4,40 

0,68 

Lab(s) used 

4,45 

0,69 

2 

Subject Domain 

4,28 

0,91 

Keyword(s) 

4,38 

0,75 

Title 

4,39 

0,86 

3 

Keyword(s) 

4,18 

1,06 

Language(s) 

4,30 

0,87 

Subject Domain 

4,22 

0,95 

4 

Lab(s) used 

4,17 

0,91 

Lab(s) used 

4,29 

0,79 

Language(s) 

4,20 

0,99 

5 

Language(s) 

4,17 

0,83 

Subject Domain 

4,21 

0,77 

Keyword(s) 

4,16 

0,90 

6 

Age Range 

4,15 

0,93 

Age Range 

4,12 

0,75 

Organizational 

Requirements 

4,09 

1,05 

7 

Educational objectives 

3,87 

1,05 

Organizational 

Requirements 

3,98 

0,97 

Age Range 

4,07 

1,00 

8 

Students’ prior knowledge 

3,76 

1,00 

Average learning 
time 

3,96 

1,02 

Educational 

objectives 

4,01 

1,00 

9 

Average learning time 

3,70 

1,01 

Students’ prior 
knowledge 

3,92 

1,03 

Status 

4,01 

1,20 
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10 

Organizational 

Requirements 

3,70 

0,98 

Additional materials 
included 

3,90 

1,03 

Additional 

materials 

4,00 

1,04 

11 

Status 

3,65 

1.17 

Educational 

objectives 

3,89 

0,89 

Students’ prior 
knowledge 

3,94 

1.11 

12 

Collaborative learning 
method used 

3,63 

1.1 1 

Status 

3,89 

1,14 

Average learning 
time 

3,93 

1,13 

13 

Additional materials 

3,63 

1,04 

Level of interaction 

3,86 

0,97 

Level of 
interaction 

3,88 

1,07 








Collaborative 



14 

Level of interaction 

3,58 

0,99 

Level of difficulty 

3,86 

1,05 

learning method 
used 

3,84 

1.12 





Collaborative 






15 

Level of difficulty 

3,57 

0,99 

learning method 
used 

3,68 

0,95 

Level of difficulty 

3,77 

1,16 

16 

Access rights 

3,43 

1.17 

Access rights 

3,43 

1,17 

Access rights 

3,65 

1.27 

17 

Owner and Contributor s) 

3,10 

1,24 

Owner and 
Contributor(s) 

3,23 

1,24 

Owner and 
Contributor s) 

3,31 

1,37 


It is worth noticing that no elements received very low score, in fact, the lowest score in average was 3.10 
- corresponding to element “Owner and Contributor(s)” (for the context of use related to general search for 
lesson plans in the Go-Lab repository), which is still on the positive side of the likert scale. Thus, an overall 
indication could be that none of the metadata elements can be regarded as non-important and our proposed 
metadata schema can be considered as valid and useful for the science teachers participated in our study. 
Nevertheless, besides this general indication, it is worth further analyzing the ranking of the metadata 
elements based on their average score for each context of use. More specifically, as we can notice from Table 
4, there are some elements which are highly ranked across all three contexts of use. These elements are (as 
presented in Table 5): (a) “Title” (1-2-1: ranked 1st for the context of use related to making a general search 
for lesson plans in the Go-Lab Repository, 1st for the context of use that was related to filtering search 
results of lesson plans in the Go-Lab Repository and 2nd for the context of use that was related to viewing 
the preview page of a lesson plan in the Go-Lab Repository), (b) “Lab(s) used” (4-4-1), (c) “Subject 
Domain” (2-5-3), (d) “Keywords” (3-2-5), (e) “Language(s)” (5-3-4) and (f) “Age Range” (6-6-7). 

Table 5. Highly Ranked Lesson Plan Metadata Elements across all three Contexts of Use 


No 

Metadata Element 

General Search 
(Average Value - 
Rank) 

Filtering 
(Average Value - 
Rank) 

Preview Page 
(Average Value - 
Rank) 

1 

Title 

4,39(1) 

4,40(1) 

4,39 (2) 

2 

Labs used 

4,17(4) 

4,29 (4) 

4,45 (1) 

3 

Subject Domain 

4,28 (2) 

4,21 (5) 

4,22 (3) 

4 

Keyword(s) 

4,18(3) 

4,38 (2) 

4,16(5) 

5 

Language(s) 

4,17(5) 

4,30 (3) 

4,20 (4) 

6 

Age Range 

4,15(6) 

4,12(6) 

4,07 (7) 


Based on these results presented in Table 5, we can identify that: 

■ Science teachers are interested in searching lesson plans using the title and the languages used in the 
lesson plans. This was an expected finding since these elements are very general and they are very 
important in any type of search performed on web based repositories of educational resources as 
highlighted by other studies (Tsourlidaki et al., 2015; Palavitsinis et al. 2009; Krull et al. 2006) 

■ Science teachers are interested in searching lesson plans that are developed around specific online 
labs that they are probably competent in using. 

■ Science teachers are interested in searching lesson plans with metadata elements that are mapped to 
the science curriculum. These elements are 3 out of the 6 most highly ranked elements (namely, 
subject domain, keywords and age range). 
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On the other hand, the elements that were ranked low across all three contexts were the following (as 
presented in Table 6): (a) “Lesson Plan Owner and Contributors” (18-18-18), (b) “Lesson Plan Access 
Rights” (17-17-17), (c) “Level of Difficulty” (16-14-16) and (d) “Level of Interaction” (15-13-13). 


Table 6. Lowest Ranked Lesson Plan Metadata Elements across all three Contexts of Use 


No 

Metadata Element 

General Search 
(Average Value - 
Rank) 

Filtering 
(Average Value - 
Rank) 

Preview Page 
(Average Value - 
Rank) 

1 

Owner and 
Contributor(s) 

3,10(18) 

3,23(18) 

3,31 (18) 

2 

Access Rights 

3,43(17) 

3,43 (17) 

3,65(17) 

3 

Level of Difficulty 

3,57(16) 

3,86(14) 

3,77(16) 

4 

Level of Interaction 

3,58(15) 

3,86(13) 

3,88(13) 


Nevertheless, the average scores (as presented in Table 6) of these metadata did not call for an automatic 
elimination from the list of metadata elements of our proposed metadata schema. Moreover, we can identify 
that it makes sense that “Lesson Plan Owner and Contributors” and “Lesson Plan Access Rights” metadata 
elements are at the bottom of the ranking list since they do not offer rich information in order to facilitate the 
selection of a certain Lesson Plan, but they are essential in order to give credits to the owner of the offered 
Lesson Plan and to clarify the access rights in order to avoid legal issues. 


6. CONCLUSIONS AND FUTURE WORK 

Building on an identified need of science teachers to be able to find and select lesson plans supported by 
RVLs, in an efficient and effective way, the main contribution of the paper is the proposal for a metadata 
schema that can be used for characterizing lesson plans supported by RVLs. Moreover, the validation of this 
metadata schema with 82 European school science teachers demonstrated (a) the added value of the proposed 
metadata schema and (b) important elements are those that describe the RVLs utilized by a lesson plan, as 
well as those metadata elements that provide a concrete mapping to the science curriculum (namely, subject 
domain, keywords and age range). 

Future work will focus on two main strands. First, we aim to collect teachers’ search logs through the 
Go-Lab repository and analyze them for validating further the significance of the metadata elements of our 
proposed metadata schema. Secondly, we aim to combine the proposed metadata schema for lesson plans 
supported by RVLs with our previous work on describing RVLs with appropriate metadata (Zervas et al., 
2014) towards developing decision support systems that will facilitate science teachers to select appropriate 
RVLs during the lesson planning process. An initial work towards this direction was reported in Zervas et al. 
(2015) and it could be significantly enhanced by incorporating the results of this study. 
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ABSTRACT 

The purpose of this study was to investigate if students are prepared to release any personal data in order to inform 
learning analytics systems. Besides the well -documented benefits of learning analytics, serious concerns and challenges 
are associated with the application of these data driven systems. Most notably, empirical evidence regarding privacy 
issues for learning analytics is next to nothing. A total of 330 university students participated in an exploratory study 
confronting them with learning analytics systems and associated issues of control of data and sharing of information. 
Findings indicate that sharing of data for educational purposes is correlated to study related constructs, usage of Internet, 
awareness of control over data, and expected benefits from a learning analytics system. Based on the relationship 
between the willingness to release personal data for learning analytics systems and various constructs closely related to 
individual characteristics of students it is concluded that students need to be equally involved when implementing 
learning analytics systems at higher education institutions. 


KEYWORDS 

Learning analytics; higher education; control over data; transparency 


1. INTRODUCTION 

Massive administrative, systems, academic, and personal data within educational settings and higher 
education institutions are becoming more and more available. These vast amounts of educational information 
provide new opportunities to improve administrative decision-making as well as to facilitate learning and 
instruction. Learning analytics uses dynamic information about learners and learning environments, 
assessing, eliciting and analyzing it, for real-time modeling, prediction, and optimization of learning 
processes, learning environments, and educational decision-making (Ifenthaler, 2015). 

Serious concerns and challenges are associated with the application of learning analytics (Pardo & 
Siemens, 2014). For instance, not all educational data is relevant and equivalent. Therefore, the validity of 
data and its analyses is critical for generating useful summative, real-time, and predictive insights 
(Macfadyen & Dawson, 2012). Furthermore, limited access to educational data may generate disadvantages 
for involved stakeholders. For example, invalid forecasts may lead to inefficient decisions and unforeseen 
problems (Ifenthaler & Widanapathirana, 2014). Moreover, ethical and privacy issues are associated with the 
use of educational data for learning analytics. That implies how personal data are collected and stored as well 
as how they are analyzed and presented to different stakeholders (Slade & Prinsloo, 2013). 

Currently, most research towards privacy issues in learning analytics refers to guidelines from other 
disciplines such as Internet security or medical environments (Pardo & Siemens, 2014). However, due to the 
contextual characteristics of privacy an adoption from other contexts is not recommendable (Nissenbaum, 
2004). More importantly, empirical evidence regarding privacy issues for learning analytics is scarce. 
Therefore, the purpose of this exploratory study was to investigate if students are willing to release any 
personal data for informing learning analytics systems. 
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2. RELEASE OF PERSONAL DATA 


2.1 Privacy in the Digital World 

The most general definition of privacy is freedom from interference or intrusion (Warren & Brandeis, 1890). 
A legal definition of the concept of privacy is a person’s right to control access to his or her personal 
information (Gonzalez, 2015). More precisely, privacy is a combination of control and limitations, which 
implies the possibility of individuals to influence the flow of their personal information and to hamper others 
to access their information (Heath, 2014). 

Within the digital world, this view on privacy seems to be no longer valid. Many individuals are willing 
to share personal information without being aware of who has access to the provided data and how the data 
will be used as well as how to control ownership of the provided data (Solove, 2004). Accordingly, data are 
generated and provided automatically through online systems which limits the control and ownership of 
personal information in the digital world (Slade & Prinsloo, 2013). Only recently, this phenomenon has been 
adopted by higher education institutions through the implementation of learning analytics. 

2.2 Privacy Principles for Learning Analytics 

Higher education institutions have always used a variety of data about students, such as socio-demographic 
information, higher education entrance qualification grades, or pass and fail rates, for academic 
decision-making as well as resource allocation (Long & Siemens, 2011; Prinsloo & Slade, 2014). Such data 
can help to successfully predict dropout rates of first-year students and implement strategies to support 
learning and instruction as well as to retain students (Tinto, 2005). 

Accordingly, advanced digital technologies and learning analytics systems enable higher education 
institutions to collect dynamic real-time data from all student activity within the higher education institutions’ 
systems which offers huge potential for personalized and adaptive learning experiences and support (Berland, 
Baker, & Bilkstein, 2014). Consequently, higher education institutions are required to address privacy issues 
linked to learning analytics: They need to define who gets access to which data, where and how long will the 
data be stored, and which procedures and algorithms are implemented to further use the available data. 

Slade and Prinsloo (2013) as well as Pardo and Siemens (2014) established several principles for privacy 
and ethics in learning analytics. They highlight the active role of students in their learning process, the 
temporary character of data, the incompleteness of data on which learning analytics are executed, 
transparency regarding data use, as well as purpose, analyzes, access, control, and ownership of the data. 
However, empirical evidence toward student perceptions of privacy principles related to learning analytics is 
lacking. 

2.3 Purpose of the Study 

Empirical research is currently addressing the validity and effectiveness of learning analytics systems for 
learning, instruction, and educational decision-making (Ali, Hatala, Gasevic, & Jovanovic, 2012; Gasevic, 
Dawson, & Siemens, 2015; Ifenthaler & Widanapathirana, 2014). In contrast, ethics and privacy issues in 
learning analytics are in an early stage of research (Pardo & Siemens, 2014). 

In this regard, the purpose of this exploratory study was to investigate if students are willing to release 
any personal data for informing learning analytics systems and if other constructs such as study interest and 
use of Internet are related. 

It is argued that first year students have to adjust to different learning and teaching requirements, manage 
workloads and course loads, as well as matching the universities’ expectations and personal interest (Bowles, 
Fisher, McPhail, Rosenstreich, & Dobson, 2014). Learning analytics may provide scaffolds to overcome the 
before mentioned hurdles especially in the first year of university studies. Specifically, we assume that 
divulging personal information within learning analytics systems is related to study related constructs such as 
year of study (Hypothesis la), course load (Hypothesis lb), and study interest (Hypothesis lc). 
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Another factor which may influence the use and acceptance of learning analytics are the students’ 
technology competencies (Kennedy, Judd, Churchward, Gray, & Krause, 2008). It is increasingly recognized 
that a majority of students possess a core set of technology-based competencies, however, no empirical 
evidence exists how these competencies influence the use and acceptance of learning analytics systems. For 
example, Trepte, Dienlin, and Reinecke (2013 ) report that students who frequently use social media tools are 
more open to disclose personal information in online environments. Therefore, we assume that releasing any 
personal data for learning analytics systems is related to the students’ percentage of use of the Internet for 
learning (Hypothesis 2a) and social media (Hypothesis 2b). 

Students’ trust and control with regard to online systems in general and learning analytics systems in 
particular may be another factor guiding the use and acceptance of learning analytics (Ennen, Stark, & 
Lassiter, 2015; Nam, 2014). We also assume that student’s willingness to provide personal data is related to 
their anticipated control over data (Hypothesis 3). 

Last, students may disclose personal data for learning analytics systems if the overall benefits for learning 
are greater than the assessed risk of releasing personal data (Culnan & Bies, 2003). We assume that releasing 
personal data for learning analytics systems is related to the anticipated benefits from a specific learning 
management system (Hypothesis 4). 


3. METHOD 


3.1 Participants and Design 

The study was designed as an online laboratory study implemented on the university’s server and conducted 
in June 2015. Participants received one credit hour for participating in the study. 

The initial dataset consisted of 333 responses. After removing incomplete responses, the final dataset 
included N = 330 valid responses (223 female, 107 male). The average age of the participants was 22.75 
years ( SD = 3.77). The majority of the participants studied in the Bachelors program (80%), with 20% of the 
participants studying in the Masters program. The average course load in the current semester was 5 courses 
(SD = 1.70). Participants reported that 33% of their Internet use was for learning, 33% was for social 
networking, 26% for entertainment, and 8% for work. 

3.2 Instruments 

3.2.1 Study Interest Questionnaire 

The study interest questionnaire (LSI; Schiefele, Krapp, Wild. & Winteler, 1993) includes 18 items 
(Schiefele, Krapp, Wild & Winteler, 1993) which focus on study-related interest such as feeling- and value- 
related valences as well as intrinsic orientation (Cronbach’s a = .90). All items were answered on a five -point 
Likert scale (1 = not at all important; 2 = not important; 3 = neither important nor unimportant; 
4 = important; 5 = very important). 

3.2.2 Technology Affinity Scale 

The technology affinity scale (TAS) focuses on information behavior to indicate educationally relevant 
activity, such as information seeking and sharing (Mills, Knezek, & Wakefield, 2013). TAS consists of 22 
items which were answered on a five-point Likert scale (1 = not at all important; 2 = not important; 
3 = neither important nor unimportant; 4 = important; 5 = very important) (Cronbach’s a = .645). 

3.2.3 Control over Data Scale 

The control over data scale (COD) focuses on access, control, and use of data in learning analytics systems, 
including four subscales: 1. Privacy of data (PLA; 5 items; Cronbach’s a = .78), 2. Transparency of data 
(TAD; 8 items; Cronbach’s a = .72), 3. Access of data (AOD; 11 items; Cronbach’s a = .83), and 4. Terms 
of agreement (TOA; 6 items; Cronbach’s a = .73). All items were answered on a five-point Likert scale 
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( 1 = not at all important; 2 = not important; 3 = neither important nor unimportant; 4 = important; 5 = very 
important). 

3.2.4 Sharing of Data Questionnaire 

The sharing of data questionnaire (SOD) focuses on specific personal information participants are willing to 
share in learning analytics systems, such as date of birth, educational history (self and parents), online 
behavior, academic performance, library usage, etc. The 28 items are answered on a Thurstone scale 
( 1 = agree, 0 = do not agree; Cronbach’s a = .74). 

3.2.5 Demographic Information 

Demographic information included age, gender, Internet usage for learning and social media, years of study, 
study major, course load, etc. 

3.3 Learning Analytics Systems 

Three different examples of learning analytics systems were presented to the participants. The first example 
was based on the Course Signals project including simple visual aids such as completion of assignments, 
participation in discussion (Pistilli & Arnold. 2010). The second example included a dashboard showing 
general information about the student, average activities over time (e.g. submissions, learning time, logins, 
interactivity), and average performance comparison across study major and university. The third example 
provided detailed insights into learning and performance including personalized content and activity 
recommendation (e.g. reading materials), self-assessments, predictive course mastery, suggestions for social 
interaction, and performance comparisons. Participants rated each of the examples regarding acceptance of 
the learning analytics system and expected benefits for learning (ALA; 10 items; Cronbach’s a = .89). 

3.4 Procedure 

Over a period of two weeks in June 2015, students were invited to participate in the laboratory study which 
included three parts. In the first part, participants received a general introduction regarding learning analytics 
and use of personal data in digital university systems. Then they completed the study interest questionnaire 
(FSI; 18 items; 8 minutes) and the technology affinity scale (TAS; 22 items; 10 minutes). In the second part, 
participants were confronted with three different learning analytics systems. After a short time to familiarize 
with each of the learning analytics system, they were asked to rate acceptance and expected use for learning 
of the learning analytics systems as well as to compare the three different systems (30 minutes). In the third 
part, participants completed the control over data scale (COD; 30 items; 20 minutes) and the sharing of data 
questionnaire (SOD; 28 items; 20 minutes). Finally, participants reported their demographic information 
( 14 items; 7 minutes). 


4. RESULTS 

Table 1 shows the zero-order correlations among the variables with regard to the first set of hypotheses. 
Students’ study year was negatively related to their course load, as was their percentage of Internet use for 
social media. Students’ study year was positively related to their percentage of Internet use for learning, as 
was their anticipated control over data. Their study interest was related to their anticipated control over data. 
Additionally, their percentage of Internet use for learning was positively related to their anticipated control 
over data as well as their expected benefits of the learning analytics system. Finally, students’ anticipated 
control over data was positively related to their expected benefits of the learning analytics system. 

A hierarchical regression analysis was used to determine whether study related variables (SY, CL, FSI), 
Internet usage (IUL, IUS), control over data (COD), and expected benefits of learning analytics systems 
(BLA) were significant predictors of sharing of data for a specific learning analytics system (SOD; 
dependent variable). Table 2 shows the four steps of entering data into the equation. The final regression 
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model explained a statistically significant amount of variance in sharing of data (SOD), A R 2 = .370, 
F(7, 329) = 28.58, p < .001. 


Table 1. Descriptives and zero-order correlations for study related variables, Internet usage variables, and data as well 

as learning analytics related variables ( N = 330) 


Variable 

1 

2 

3 

4 

5 

6 

7 

1. Study year (SY) 

- 







2. Course load (CL) 

- 378*** 

- 






3. Study interest (FSI) 

-.008 

.071 

- 





4. Internet use for learning 

.123* 

-.076 

.014 

- 




(IUL) 

5. Internet use for social media 

-.156** 

.023 

-.066 

-.032 




(IUS) 

6 . Control over data (COD) 


-.038 

. 111 * 

290*** 

.007 



7. Benefits of learning analytics 
system (BLA) 

.076 

-.017 

-.009 

.630*** 

-.006 

.362*** 

- 

M 

3.58 

5.36 

2.99 

35.00 

32.95 

2.71 

3.13 

SD 

2.30 

1.70 

.28 

21.21 

20.43 

.39 

.97 


Note. * p < .05, ** p < .01, *** p < .001 


Table 2. Regression analysis predicting sharing of data on study related variables, Internet usage, control over data, and 

expected benefits of learning analytics systems (N = 330) 



R 2 

A R 2 

B 

SEB 

P 

Step 1 

.038 

.029 




Study year (SY) 



.538 

.170 

.186** 

Course load (CL) 



-.081 

.231 

.726 

Study interest (FSI) 



-.094 

1.295 

.942 

Step 2 

.322 

.311 




Study year (SY) 



.432 

.145 

149** 

Course load (CL) 



.010 

.195 

.002 

Study interest (FSI) 



-.127 

1.093 

-.005 

Internet use for learning (IUL) 



.165 

.014 

525 *** 

Internet use for social media (IUS) 



.040 

.015 

122 ** 

Step 3 

.352 

.340 




Study year (SY) 



.366 

.143 

.127* 

Course load (CL) 



-.005 

.191 

-.001 

Study interest (FSI) 



-.609 

1.077 

-.026 

Internet use for learning (IUL) 



.149 

.015 

.474*** 

Internet use for social media (IUS) 



.037 

.015 

.114* 

Control over data (COD) 



3.144 

.806 

185*** 

Step 4 

.383 

.370 




Study year (SY) 



.373 

.140 

129** 

Course load (CL) 



-.035 

.186 

-.009 

Study interest (FSI) 



-.376 

1.054 

-.016 

Internet use for learning (IUL) 



.106 

.018 


Internet use for social media (IUS) 



.037 

.014 

.113* 

Control over data (COD) 



2.339 

.813 

.138** 

Benefits of learning analytics system (BLA) 



1.606 

.400 

234 *** 


Note. * p < .05, ** p < .01, *** p < .001 


Specifically, students’ study year (SY) positively predicted their willingness to share personal data for a 
specific learning analytics system (SOD), indicating that the higher the study year (SY), the higher the 
students’ liberality to provide personal data for educational purposes. Accordingly, Hypothesis la is 
accepted. 
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The percentage of Internet usage for learning (IUL) and social media (IUS) positively predicted the 
students’ release of personal data for learning analytics purposes (SOD), indicating the higher the usage of 
the Internet for learning and social media, the higher their disposition to share personal data for learning 
analytics systems. Hence, Hypotheses 2a and 2b are accepted. 

The student’ awareness about control of data (COD) positively predicted their preparedness to share 
personal data for a specific learning analytics system (SOD), indicating that the higher the awareness about 
the control of personal data, the higher their disposition to share personal data for learning analytics systems. 
Thus, Hypothesis 3 is accepted. 

The expected benefits of a learning analytics system (BLA) positively predicted the students’ release of 
personal data for learning analytics purposes (SOD), indicating the higher the expected benefit of the learning 
analytics system, the higher the readiness to provide personal data for learning analytics purposes. 
Consequently, Hypothesis 4 is accepted. As shown in Table 2, no significant correlations were found for 
course load (CL) and study interest (FSI). So, Hypotheses lb and lc are rejected. 


5. DISCUSSION 

At a time of growing interest in learning analytics systems of higher education institutions, it is important to 
understand the implications of privacy principles to ensure that implemented systems are able to facilitate 
learning, instruction, and academic decision-making and do not impair students perceptions of privacy. To a 
large extend, students are the producers of data used in learning analytics systems, however, passive 
recipients of information provided in dashboards (Prinsloo & Slade, 2014). 

The findings of this exploratory study highlight an overall interest of students in learning analytics 
systems. As students mature in their higher education studies they seem to be more aware of the context of 
sharing educational data (Bailey, Ifenthaler, Gosper, Kretzschmar, & Ware, 2015). To make the benefits of 
learning analytics and emphasize the need of sharing data within the learning analytics system to first year 
students, tutoring systems and/or training sessions need to be implemented accordingly. 

Students spend a large amount of time for using the Internet for learning and social media activities. Not 
surprisingly, spending time on the Internet is associated with the openness of sharing data for learning 
analytics systems. This effect may be explained by trust students generate with regard to online systems in 
general and learning analytics systems in particular (Ennen et al., 2015). The relationships between perceived 
control over personal data and expected benefits as well as sharing personal data is closely related to the 
phenomenon of trust (Nam, 2014). These findings indicate that a high computer literacy is prerequisite for 
the acceptance of learning analytics as well as the willingness to share data and should be systematically 
trained. 

From a holistic point of view, learning analytics may provide multiple benefits for higher education 
institutions and for involved stakeholders and different data analytics strategies can be applied to produce 
summative, real-time and predictive insights (Ifenthaler, 2015). For example, students may use summative 
learning analytics implemented as an interactive dashboard to analyze learning outcomes of individual 
courses after completing a semester of study or track their progress towards self-defined goals (e.g., credit 
points). Students may also be able to compare their own learning paths and outcomes between individual 
units or courses. This may enable students to understand their learning habits and to adjust their learning 
strategies as well as private habits in order to be successful in their studies. On the same dashboard or within 
a learning management system, students may receive real-time learning analytics information based on their 
currently available data. Automated interventions may point them to learning materials and tips for 
progressing further in a particular study unit. Students may take self-assessment on a specific topic and 
receive just-in-time feedback or get recommendations to participate in online discussions or connect to peers 
using preferred social media. Predictive learning analytics for students may help to optimize the learning 
path in a specific study unit by providing them probabilities of success when choosing a particular pathway. 
Such predictions are expected to increase the overall engagement and success rates of students (Ifenthaler, 
2015). 

However, reliable and valid learning analytics systems require rich and current information of students 
including personal characteristics and preferences, academic performance, educational pathways and logfiles 
of various online learning systems. If the underlying learning analytics algorithms do not have access to the 
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required information, the above-described benefits cannot be produced. While higher education institutions 
implement learning analytics systems, students may find themselves in a dilemma situation concerning the 
divulgence of personal information for learning analytic systems. In order to overcome such a dilemma 
situation, it is necessary to provide students transparency of the implemented learning analytics system and 
its underlying algorithms, as well as clear guidelines towards access, analysis, control, ownership and use of 
relevant data. 


6. CONCLUSION 

Remaining questions such as who should get access to which data, where and how long will the data be 
stored, which analyzes and deductions are conducted and are the students aware of the data collected from 
them need to be discussed in prospective research. From an instructional design point of view, research may 
focus on usability, personalization, and adaptivity of learning analytics systems. Understanding these factors 
may be crucial for implementing learning analytics systems at higher education institutions. Student’s 
computer literacy is expected to be a prerequisite for using learning analytics systems. Professional 
development for learning analytics including transparency of underlying algorithms and involving all 
relevant stakeholders in the development and implementation phases may help to increase trust and 
acceptance in the systems. 

Students are more than shattered bits of information given and produced while interacting with learning 
analytics systems implemented by higher education institutions (Solove, 2004). Learning analytics may 
reveal personal information and insights into an individual learning history, however, they are not accredited 
and far from being unbiased, comprehensive, and valid. 


REFERENCES 

Ali, L., Elatala, M., Gasevic, D., & Jovanovic, J. (2012). A qualitative evaluation of evolution of a learning analytics tool. 
Computers & Education, 58(1), 470-489. 

Bailey, M„ Ifenthaler, D., Gosper, M., Kretzschmar, M., & Ware, C. (2015). The changing importance of factors 
influencing students’ choice of study mode Technology, Knowledge and Learning, 20(2), 169-184. doi: 
10. 1007/s 10758-015-9253-9 

Berland, M„ Baker, R. S., & Bilkstein, P. (2014). Educational data mining and learning analytics: Applications to 
constructionist research. Technology, Knowledge and Learning, 79(1-2), 205-220. doi: 10.1007/sl0758-014-9223-7 

Bowles, A., Fisher, R., McPhail, R., Rosenstreich, D., & Dobson, A. (2014). Staying the distance: students" perception of 
enablers of transition to higher education. Higher Education Research & Development, 33(2), 212-225. 

Culnan, M. J., & Bies, R. J. (2003). Consumer privacy: Balancing economic and justice considerations. Journal of Social 
Issues, 59(2), 323-342. doi: 10.1111/1540-4560.00067 

Ennen, N. L„ Stark, E., & Lassiter, A. (2015). The importance of trust for satisfaction, motivation, and academic 
performance in student learning groups. Social Psychology of Education, doi: 10. 1007/s 1 121 8-01 5-9306-x 

Gasevic, D„ Dawson, S., & Siemens, G. (2015). Let’s not forget: Learning analytics are about learning. TechTrends, 
59(1), 64-71. doi: 10.1007/sll528-014-0822-x 

Gonzalez, D. (2015). Managing online risk: Apps, mobile, and social media security. Oxford: Elsevier. 

Eleath, J. (2014). Contemporary privacy theory contributions to learning analytics. Journal of Learning Analytics, 7(1), 
140-149. 

Ifenthaler, D. (2015). Learning analytics. In J. M. Spector (Ed.), The SAGE encyclopedia of educational technology (Vol. 
2, pp. 447^-51). Thousand Oaks, CA: Sage. 

Ifenthaler, D.. & Widanapathirana, C. (2014). Development and validation of a learning analytics framework: Two case 
studies using support vector machines. Technology, Knowledge and Learning, 79(1-2), 221-240. doi: 
10. 1007/s 10758-014-9226-4 

Kennedy, G., Judd, T., Churchward, A., Gray, K., & Krause, K. (2008). First year students’ experiences with technology: 
are they really digital natives? Australasian Journal of Educational Technology, 24(1), 108-122. 

Long, P. D.. & Siemens, G. (2011). Penetrating the fog: Analytics in learning and education. EDUCAUSE Review, 46(5), 
31-40. 


17 



ISBN: 978-989-8533-43-2 ©2015 


Macfadyen, L., & Dawson, S. (2012). Numbers are not enough. Why e-Learning analytics failed to inform an 
institutional strategic plan. Educational Technology & Society, 15(3), 149-163. 

Mills, L. A.. Knezek, G. A., & Wakefield, J. S. (2013). Understanding information seeking behavior in technology 
pervasive learning environments of the 21st century. Turkish Online Journal of Educational Technology, 72(4), 200- 
208. 

Nam, C. W. (2014). The effects of trust and constructive controversy on student achievement and attitude in online 
cooperative learning environments. Computers in Human Behavior, 37, 237-248. doi: 10. 1016/j.chb. 2014.05. 007 
Nissenbaum. H. (2004). Privacy as contextual integrity. Washington Law Review, 79(1), 1 19-157. 

Pardo, A., & Siemens, G. (2014). Ethical and privacy principles for learning analytics. British Journal of Educational 
Technology, doi: 10. 1 1 1 1/bjet. 12152 

Pistilli. M. D., & Arnold, K. E. (2010). Purdue Signals: Mining real-time academic data to enhance student success. 

About campus: Enriching the student learning experience, 75(3), 22-24. 

Prinsloo, P„ & Slade, S. (2014). Student data privacy and institutional accountability in an age of surveillance. In M. E. 
Menon, D. G. Terkla, & P. Gibbs (Eds.), Using data to improve higher education. Research, policy and practice (pp. 
197-214). Rotterdam: Sense Publishers. 

Schiefele, U„ Krapp, A., Wild, K. P„ & Winteler, A. (1993). Der „Fragebogen zum Studieninteresse“ (FSI). Diagnostica, 
39(A), 335-351. 

Slade, S., & Prinsloo, P. (2013). Learning analytics: Ethical issues and dilemmas. American Behavioral Scientist, 57(10), 
1510-1529. doi: 10.1177/0002764213479366 

Solove, D. J. (2004). The digital person: Technology and privacy in the information age. New York, NY: New York 
University Press. 

Tinto, V. (2005). Reflections on student retention and persistence: moving to a theory of institutional action on behalf of 
student success. Studies in Learning, Evaluation, Innovation and Developmental Psychology, 2(3), 89-97. 

Trepte, S.. Dienlin, T., & Reinecke, L. (2013). Privacy, self-diclosure, social support, and social network site use. 
Stuttgart: University of Hohenheim. 

Warren, S. D., & Brandeis, L. D. (1890). The right to privacy. Harvard Law Review, 4(5), 193-220. 


18 



12th International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2015) 


TAGALONG: INFORMAL LEARNING FROM 
A REMOTE COMPANION WITH MOBILE PERSPECTIVE 

SHARING 


Scott W. Greenwald, Mina Khan*, Christian D. Vazquez* and Pattie Maes 

MIT Media Lab, Cambridge, MA 02139, USA 


ABSTRACT 

Questions often arise spontaneously in a curious mind, due to an observation about a new or unknown environment. 
When an expert is right there, prepared to engage in dialog, this curiosity can be harnessed and converted into highly 
effective, intrinsically motivated learning. This paper investigates how this kind of situated informal learning can be 
realized in real-world settings with wearable technologies and the support of a remote learning companion. In particular, 
we seek to understand how the use of different multimedia communication mediums impacts the quality of the interaction 
with a remote teacher, and how these remote interactions compare with face-to-face, co-present learning. A prototype 
system called TagAlong was developed with attention to features that facilitate dialog based on the visual environment. It 
was developed to work robustly in the wild, depending only on widely-available components and infrastructure. A pilot 
study was performed to learn about what characteristics are most important for successful interactions, as a basis for 
further system development and a future full-scale study. We conclude that it is critical for system design to be informed 
by (i) an analysis of the attentional burdens imposed by the system on both wearer and companion and (ii) a knowledge 
of the strengths and weaknesses of co-present learning. 


KEYWORDS 

Informal learning, situated learning, remote learning, contextual memory, wearable technology 


1. INTRODUCTION 

Much of the practical and tacit everyday knowledge employed in workplaces is acquired on the job, through 
learning-by-doing. One reason why this type of learning is highly effective is that knowledgeable coworkers 
can be found nearby to not only assist in completing tasks at hand, but also to subsequently engage in broader 
discourse, in which ideas from specific tasks are generalized and abstracted. That is, they use these task 
examples as props to explain ideas that generalize to other tasks, explain the reasons behind procedures, and 
so on. The physical elements at hand provide a point of reference for the learner to become curious and ask 
follow-up questions, based not only on what the expert has chosen to highlight, but also on her independent 
observations of the environment and apparatus. With a shared physical environment as a precondition, this 
type of learning episode requires three attributes: the immediate need for assistance as impetus to start the 
conversation, an expert available to assist, and the opportunity to seamlessly transition from an 
assistance-focused dialog to a broader discussion through which deep knowledge exchange can happen. 

We seek to broaden the applicability of this powerful episodic learning model in two ways. Firstly, we 
seek to make it possible to learn in this way from an expert who is remote instead of co-present. This would 
dramatically increase the reach of such interactions, so they could happen at any time in any physical or 
geographic location. Secondly, we aim to apply this model not just to assistance and learning in workplaces 
or communities of practice, but also to the myriad other contexts where curiosity may arise, not only out of 
necessity. In general, this means engaging in dialog to answer and expound on questions that arise due to the 
immediate physical surroundings. By supporting informal and exploratory learning in this way, we can move 
towards a world where deep (human) learning can happen anywhere and everywhere, driven by the intrinsic 
motivation of curiosity. 


These authors contributed equally to this work. 
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In this paper, we seek to investigate specifically what kind of system is necessary to facilitate a fluid 
dialog between learner and remote expert, allowing each to make reference to specific physical objects in the 
learner’s environment? Can such a system be made effective without being cumbersome or requiring 
distracting device interactions? Can this be achieved today, building only on readily available devices and 
infrastructure? This represents an exploratory phase of research, the goal of which is to inform the design of 
future systems, as well as identify specific questions for future research, including full-scale studies that 
employ such systems. 

We designed a prototype system called TagAlong and performed a qualitative pilot study in which the 
same task (discussing fine artwork) was performed with three different communication mediums: (i) 
co-present (oral and gestural) communication, (ii) mobile phone-based video chat, and (iii) using the 
TagAlong prototype system, utilizing synchronous audio in combination with still image capture and 
annotation. 

This paper is structured as follows: Section 2 discusses prior work and positions TagAlong within this 
framework. This allows us to build on what is already known, as well as highlight the challenges that are 
particular to our use case. Then, Section 3 presents the design goals for the TagAlong prototype system, 
initial design, and enhancements based on the results of preliminary testing. Section 4 describes a pilot study 
based on a learning scenario in an art museum. Section 5 talks about implications for future work, and 
finally. Section 6 presents concluding remarks. 


2. RELATED WORK 

This work is situated at the intersection of learning, communication technology and wearable technology. 
From a learning perspective, the theoretical underpinnings are based on the concepts of informal learning, 
situated learning and contextual memory. Informal learning happens through curiosity or necessity within a 
social or experiential context, and is unintentional from the perspective of the learner. Situated learning refers 
to the acquisition of knowledge relevant to needs or actions at hand. In the field of language learning it has 
been shown that students are more receptive to learning relevant vocabulary and phrases in such 
circumstances (Brown 1987). Closely related is the theory of contextual memory, which holds that a person 
is more likely to recall information when situated in a context analogous to the one where they were 
originally exposed to it. This is explained by the presence of similar memory cues both at the time of 
exposure and the time of retrieval (Tulving and Thomson 1973; Davies and Thomson 1988). 

On the communication technology side some research has focused on use cases for collaboration with 
shared subject matter, evaluating the usefulness of different communication capabilities and mediums. For 
instance, Chastine et al. (Chastine et al. 2007 ) have looked at different configurations of physical and virtual 
object representations in a collaborative 3D task, and investigated the impact of these representations in a 
virtual environment. They found that a fundamental requirement is the ability for users to effectively refer to 
artifacts within the shared environment. Ochsman et al. (Ochsman and Chapanis 1974) have investigated the 
comparative value of audio, video, and text channels to support cooperative problem solving, concluding that 
the audio channel is the most critical medium. 

Other research has focused on audio/video conferencing use cases where the communication medium 
primarily carries voice and video of the participants themselves. Isaacs et al. (Isaacs and Tang 1994) 
evaluated video conferencing as compared to audio calling and concluded that there is significant social 
value to seeing those you interact with, especially when the purpose is professional team-building. It 
facilitates interpreting non-verbal information, noticing peripheral cues and expressing attitudes. 

Even though video offers a rich communication medium and provides several advantages for interpreting 
non-verbal information between collaborators when using desktop or personal computers, in the case of task 
assistance or situated learning via a wearable devices, this might not be the case. For example, video of the 
face of the user using a wearable device would be both difficult to capture and not add much value when the 
purpose is to communicate about the environment. Careful attention must be given to what behaviors are 
supported through the choice of medium and other affordances of the system. 

Next we consider related work in wearable device interaction including assistance systems and 
telepresence. The development of wearable computing was just burgeoning when a seminal work carried out 
by Starner et al. (Starner et al. 1997) put forth the concept of computing that proactively assists a user. Early 
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work such as that carried out by Feiner et al. (Feiner et al. 1997) looked specifically at how information could 
be presented to users with wearable computers - highlighting the fact that the type of interactions required by 
such systems are completely different from the ones required by desktops or mainframe computers. 

Thereafter, researchers began to investigate how wearable systems could be used to facilitate remote 
collaboration since, by their nature, they were able to track and convey information about the wearer’s 
surrounding environment. For example, early work carried out by Mann (S. Mann 2000) shows that a mobile 
system that gives very simple feedback in the wearer’s environment (a laser dot) can effectively be used to 
experience visual collaborative telepresence. Among the disadvantages of this system are that (i) it is limited 
to very simple feedback about the environment, (ii) this feedback is highly ephemeral, since any movement 
by either wearer or companion results in moving the laser dot off target, and (iii) it is quite obtrusive, if not 
dangerous, to those individuals in the immediate surroundings of the system due its utilization of a laser 
pointer. Building on the TelePointer concept, Gurevich et al. (Gurevich et al. 2012) developed a system for 
more sophisticated feedback called TeleAdvisor. TeleAdvisor is a stationary system that gives visual 
feedback using a projector mounted on a robotic arm. It enables a remote helper to view and interact with the 
workers’ workspace, while controlling the point of view. 


3. TAGALONG SYSTEM 

The TagAlong system is a mobile context-sharing system that runs on Google Glass and a mobile phone or 
tablet. The system wearer can send still images on-demand to a remote companion or teacher, who can then 
reply by annotating the source material and sending it back. Synchronous interaction is optionally supported 
through the use of a real-time audio channel. In this section we describe the design goals, interaction design, 
and architectural decisions that make the system work robustly in the wild. 

3.1 Design Goals 

As outlined above, the TagAlong prototype system is intended to be both well-adapted to facilitating dialog 
about the wearer’s physical surroundings and usable in everyday settings. The latter requirement goes well 
beyond what is necessary just to perform our experiment. Designing and building the system to be usable in 
everyday settings makes our experimental results significant when considering (i) what kinds of systems can 
be built into an everyday usage flow, without assuming mass adoption, and (ii) what device technology and 
infrastructure are already here today. This way we can comment on how this will change in the immediate 
term, as well as where it is most important to invest effort in both of these areas. Accordingly, our design 
goals are as follows: 

• Create a system that can be worn continuously and with minimal burden by the wearer, and can allow 
both users (the wearer and the companion) interact with the system in a mobile setting. 

• On the wearer side, the system should operate in the background requiring little of her attention. Giving 
input to the system should incur little startup cost, and when information from companion wearer 
arrives, it should be noticeable but not disruptive. 

• Support synchronous interaction, so as to reap maximum benefit from high -engagement interactions 
with the companion. 

• Additionally, support asynchronous interactions. There is a spectrum from low-engagement, 
low-bandwidth communication to high -engagement, high-bandwidth communication that needs to be 
supported for in-the-wild usage. 

• Allowing anyone with a smartphone to play the role of the companion, and accordingly develop a flow 
for initiating interactions that is natural and requires a minimal amount of effort. This dramatically 
increases the reach of real-world usage of the prototype system. 
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3.2 Interaction Design 


There are two user roles in the TagAlong system: (1) the wearer, who uses a Google Glass connected via 
Bluetooth to an Android mobile device, (2) the companion, who is any person with a smartphone web 
browser that the wearer connects with, as shown in Figure 1. 

When the wearer wishes to interact with a remote companion, he uses the app to send a text message to 
the desired companion to ask her to “connect”, or “opt-in” to a TagAlong session. As such, the session 
through which the exchange of visual information happens is asynchronous and can last indefinitely (hours, 
days, or weeks). The text message contains a link which, when clicked, opens a page that describes what the 
companion is opting in to, and then allows her to confirm. Figure 2 shows the registration and connection 
process for the TagAlong users. For interactions with synchronous audio, the wearer can call the companion 
using the carrier network. 

When the wearer wishes to take action by sending an image, she must press a button once to enter a 
view-finder mode that opens a camera preview so that the wearer may take a picture using Google Glass. The 
wearer presses a button to take a picture, and the companion receives a text message containing a url. 
Clicking on the url causes opens up the image in a browser window and allows the companion to annotate 
and edit the image. The companion can then send back the annotated image, which is displayed on the 
wearer’s device and accompanied by an audio chime. 



Figure 1. Basic Information Flow in TagAlong System 


Figure 2. Interaction Initiation Flow 


3.3 TagAlong Wearer UI Enhancements 

Exploratory usage of the initial implementation of TagAlong exposed a number of usability problems in the 
wearer interface. We describe these problems and the features we introduced to address them. 

3.3.1 Four-button Input 

We found the native Glass touchpad to be both unreliable at detecting input events, as well as awkward to 
operate in public settings (this awkwardness was heightened by the unreliability of event detection, when 
repeated attempts needed to be made). For this reason we used a wireless slide changer remote as an input 
device. This device is unobtrusive, can be easily stowed in a pocket or purse, and offers tactile feedback to 
support eyes-free operation. 

3.3.2 Status Notifications 

In early trials of our system, a lack of feedback for the wearer left them uncertain of the state of the system. 
We had used audio chimes to notify the user of progress - such as an image being captured aor successfully 
uploaded to the server. In moderately noisy environments or with moderate attentional loads, it was easy to 
miss these updates. Our enhanced design uses visual status indicators to indicating whether a message has 
been sent and seen by the receiver. In exploratory trials, users reported better usability when status messages 
were used. 
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3.3.3 Viewfinder 

Another important user interface choice was to use a viewfinder. The canonical use of the Google Glass 
camera takes a picture without showing a live camera preview. In our exploratory trials, users complained 
that they could not frame the picture sent to the remote companion, and this limited the effectiveness of these 
pictures at communicating the object of their attention. Introducing a viewfinder allows the user to know 
immediately both that the device is attempting to take a picture, and that the field of view is scoped and 
aligned as desired. 


4. PILOT STUDY: LEARNING FROM AN EXPERT 

The goal of this study was to understand how the fully-mobile TagAlong system compares with video 
streaming and face-to-face communication in terms of effectiveness at communicating about the visual 
environment for learning purposes. The basic activity in the study is an informal learning dialog about a work 
of art between an art expert and a novice. 

4.1 Setup 

A remote art expert interacts informally with a non -expert to convey knowledge about a specific work of art, 
which the non-expert is in the presence of. No guidance is given to either party about how to initiate the 
dialog. They interact using one of three conditions (i) TagAlong with live audio, (ii) video streaming on a 
mobile phone with live audio, and (iii) co-present, face-to-face interaction. In the dialog that they have, either 
party can determine the subject, questions can be asked, and clarifications requested. 



Figure 3. Wearer in Pilot Study Figure 4. Companion in Pilot Study 


In the TagAlong system condition, participants are connected with a live audio stream, and can use the 
system to exchange images and annotations of the subject matter in the wearer’s environment as shown in 
Figure 3 and Figure 4. The wearer’s interaction is hands-optional, since she only needs to use her hands when 
she wants to send an image. No physical posture change is required for her to shift focus between the 
system’s visual feedback and the world, and the input device can be operated eyes-free. 

In the second condition, the “wearer” streams video from a hand-held mobile phone. The wearer and 
companion are once again connected with a live audio stream. The video stream of the wearer’s rear-facing 
phone camera is previewed on-screen and streamed real-time to the companion’s mobile phone. There is no 
affordance for spatial annotation in the interface, but the wearer can physically point with his free hand to 
indicate an object of interest. 

In the third condition, learner and expert are co-present and communicate face-to-face. The two stand 
next to each other in front of the painting and discuss. 
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4.2 Procedure 

We tested a total of four wearers, each with one of two art expert companions. The wearers were each 
involved in two trials, and in each trial, the wearer and companion would discuss two works of art for five 
minutes each. For each wearer, one of the two trials was TagAlong, and for the other trial, two participants 
tried video streaming, while the other two tried face-to-face explanation. For each expert, the four art pieces 
stayed the same across trials, but the order of the trial conditions was switched in subsequent trials with 
respect to the TagAlong trial. 

After both trials were completed, an in-depth interview lasting between 5 and 15 minutes was conducted 
with both wearer and companion present. In addition, after each expert’s second trial, a solo follow-up 
interview was performed, where she was asked to contrast the experience of explaining the same work of art 
in the two different conditions, considering each work of art individually. 

4.3 Results 

First we compare results from the TagAlong system with video streaming, and then with the co-present 
learning condition. 

4.3.1 Still Images vs. Streaming Video 

Still images and streaming video each presented their own advantages and disadvantages. Still images from 
TagAlong provided a clear advantage over streaming video as a vehicle for detailed and persistent 
annotations by the companion. The companion could circle, underline, outline, and label with text specific 
visual elements. In the video streaming condition, companions needed to use verbal cues and gestures to 
draw the wearer’s attention to particular subject matter in order to explicate it. Once a subject was identified, 
a verbal description was needed to make detailed comments. As a corollary, the ability to freeze a subject 
matter allowed the dialog to be more focused. With moving images, the companion felt she had to follow the 
dynamic whim of the wearer, whereas with still images, her highlighting specific details also caused the 
wearer to stay still long enough to focus on those details. 

A disadvantage of still images in our implementation was that the companion was limited to annotating 
only the most recent image taken by the wearer. When this was a close-up, she no longer had an effective 
way of suggesting the next subject of focus. One way of addressing this limitation would be to allow the 
companion to refer back to previous (less close-up) images to suggest the next focal point (e.g. as shown by 
Greenwald, et al (Greenwald, Khan, and Maes 2015)). 

Video streaming was clearly advantageous over still image exchange in terms of responsiveness. One 
wearer commented about using video-streaming when discussing the sculpture: 

I could show her what 1 was talking about it real-time. It was a smoother process than taking 
pictures all the time. 

That is, when the subject was a sculpture, the wearer needed to move around to find interesting 
viewpoints, and having a live stream was advantageous. 

Even so, there was agreement among participants that which system was better depended on subject 
matter. As one art expert participant expressed: 

For Matta [painting] I prefer cards [still images]. I could actually point out the things that I 
thought were interesting ... [for the sculpture] I liked video actually, I think that has a lot to do 
with the subject matter. 

That is, when the subject was a painting with different objects and details to be highlighted, the ability to 
circle and point at them was more important. 

We propose that the interfaces we experimented with are best understood as performing two separate but 
overlapping functions - first, giving the companion some level of situational awareness about the physical 
surroundings and focus of attention of the wearer, and second, creating a locus of attention that is shared and 
can be modified by one party or the other. A video feed is constantly being moved by the wearer and gives 
the companion no control over the shared locus of attention other than to intervene verbally to highlight 
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visual landmarks. In the case of TagAlong, the wearer selects the frame, and the companion can select a 
subregion and annotate by sketching. 

4.3.2 Co-Present vs Remote Learning 

One assumption we made going into this study was that “being there”, i.e., co-present learning, is always the 
best. Perhaps the most unexpected insight we gained was that this is not necessarily the case. Indeed 
participants in the face-to-face condition noted the comparatively more natural and intuitive way that gestures 
could be used for both pointing and expression. However, they highlighted that using mobile devices to 
interact remotely allowed them to focus more exclusively on the work of art. It seems that this is related to 
the social burden of face-to-face interaction, or the need to “entertain” as one participant described it. 

I felt a little more like I was watching TV, with the Glass on, drowning into this painting while 
she’s talking. But there [in person] it felt more like I was trying to entertain, hold a 
conversation, smile, get a laugh out of you [addressing companion]. 

Face-to-face interaction carries with it the burden of proxemics- the ensemble of body and facial gestures 
and eye contact that must be constantly maintained during co-present social interaction. Eliminating that 
burden liberates the learner’s attention to focus only on the subject matter. Participants, including experts and 
wearers alike, consistently echoed the sentiment that they were highly focused during the TagAlong system 
interaction in comparison to the feeling of having many distractions while the face-to-face discussion was 
taking place. This seems to support the claim that the maximum amount of attention was available for the 
artwork itself in the hands-free, remote condition (TagAlong). 

These results show that co-presence is an important point of reference which we can use to understand 
and predict what will work well. We can frame future work in terms of imitating co-presence in a targeted 
way. For example, we do wish to emulate the ability of either party to draw attention to a point in the 
environment. We do not wish to emulate the attentional burden of face-to-face social interaction. The 
generalization is that co-presence provides a wonderful set of affordances for two people to communicate in 
high-bandwidth. What it does not do is allow us to selectively switch off some of those affordances in order 
to achieve greater focus for specific tasks. In essence, future work in this area concerns identifying ways of 
learning that are better than co-present, face-to-face interactions within certain contexts or with certain 
specific purposes in mind. 


5. FUTURE WORK 

The above results point towards some specific improvements to TagAlong-like systems in the immediate 
term, as well as some challenging ones for the longer term. 

There is a straightforward concept for designing a system that imparts the powerful feeling of 
synchronous visual presence that we saw with video streaming, but also affords the important ability to 
annotate specific objects which we saw when using still images. In a hybrid system, this would be to have 
both live streaming and annotation at the same time. A split screen, or swappable picture-in-picture interface 
could be used to maintain both real-time awareness, and the ability for the companion to suggest or define a 
locus of attention. 

Although the wearer is able to “point” by framing a still image and speaking over it, the ability to engage 
more directly in a dialog of annotation with the companion is something that would be sure to add expressive 
power for the wearer and hence make dialogues richer. The challenge would be to maintain the same low 
level of attention required for operating the system. Some candidate input methods would be Live Trace 
(Colafo et al. 2013), which uses a depth sensor to allow the wearer to lasso environmental objects using a 
gesture at arms length in front of the face; the Nod ring, which uses a ring-mounted IMU to create 2D or 3D 
input signals from free hand movements, and the Thalmic Labs Myo, which interprets a small discrete set of 
hand gestures, in addition to including an IMU that could be used similarly to the Nod. 

Our results also pointed to the need for the companion to maintain the broadest possible representation of 
the environment, so that he can highlight subjects that are not currently being attended to by the wearer. This 
may be done by compiling all the data explicitly sent by the wearer device, but could also in general include 
reference information that could be externally retrieved. For example, in the case of the art museum, the 
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companion could be provided with a map of the museum, as well as high -resolution representations of all the 
art within it. With additional information, the expert companion would not be limited to just the 
comparatively low-quality images provided by the wearer. The latter would only be used to create situation 
awareness for the companion about what the wearer is currently attending to. In another use case, like 
navigating streets, maps archives like Google Maps may be used to invoke to points of reference that haven’t 
yet been visited by the wearer. 

The overarching challenge in all of this future work is to avoid confusion or attentional burden when 
making systems with these hybrid assemblages of content which is streaming and frozen, past or present, 
overlaid or peripheral, from internal and external sources, and so on. 


6. CONCLUSION 

In the present work we have demonstrated that individual mediums for learning-focused discourse have 
advantages and disadvantages based on what the subject matter is, and who is taking part in the dialog. Still 
images are good for making detailed reference to static elements of the environment, as we saw with the 
example of paintings. Video streaming, on the other hand, appears better when physical movement is 
important to the exploratory activity - in our example, viewing sculptures from different angles. Considering 
co-present versus remote teaching, in some cases face-to-face behaviors of like gesturing and making eye 
contact are helpful, but in other cases they can distract from the subject matter. In summary, our results 
support the claim that, in order to maximize the effectiveness of remote interactions, a carefully assembled 
melange of mediums should be selected for particular use cases. 

Moving forward along this path, we envision a world where seeking input from a remote expert will be as 
easy as tapping an office colleague on the shoulder. Tomorrow’s TagAlong-like systems will utilize 
numerous technologies to create ever more vivid glimpses into remote environments and the state of the 
those who occupy them. High -resolution 3D capture and display will make the environments seem real. 
Real-time computer vision applied to these data streams will make it possible for the companion to identify 
and annotate environmental objects in a way that is fast and persistent. Labels and annotations can adapt to 
changes in the environment. Input and output may take many non-visual forms. For instance, remote sports 
instruction might use EMG data to inform the companion how the wearer is moving, and muscle stimulation 
allows her intervene with correct motions. In the present work we haven’t even scratched the surface of more 
exotic forms of input and output, such as those just mentioned, and the challenges we encountered will be 
compounded when these are brought into the mix. On one hand this means fruitful grounds for future 
research, and on the other it calls for a principled approach, since we will otherwise be overwhelmed by the 
combinatorial complexity of the design space, and corresponding difficulty of finding good designs. 
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ABSTRACT 

This papers aims at analyzing the perception of Accounting professors about the necessary qualifications in Accounting 
undergraduate courses. The contribution of this study is to theoretically discuss the education of Accounting professors, 
with empirical data, because Accounting teaching requires specific competencies in the digital area. The research is 
applied in the Accounting Higher Education, descriptive in relation to the objectives, a qualitative approach to the 
problem with quantitative elements for descriptive data analysis. The technical procedure was the survey by online 
questionnaire via Google Forms. The research population are professors of the Accounting professional education in 
undergraduate courses in Brazil. The sample consisted of 378 professors, selected by the representativeness and typicality 
criteria. The results evidenced the perception of professors about the complexity of Accounting teaching, by applying a 
set of professional, academic and pedagogical competencies as well as practical knowledge about the accounting practice 
and teaching. However, they indicated the prevalence attributed by professors to the technical competency and 
professional practice knowledge over the other educational competencies. The professors showed feeling of deficiency in 
relation to the academic education received and they worried about the need for continuous training. They also prove, 
with empirical data, that mastering contents by Accounting professors is not enough to the teaching practice in 
Accounting courses. The results obtained in this paper may support institutional policies regarding continuous education 
of professors in Accounting undergraduate courses as well as the proposition of initiatives to develop them systematically 
by the education institutions. 


KEYWORDS 

Accounting Professor. Accounting Education. Professor education. Teaching Competencies. 


1. INTRODUCTION 

The education of professors in contemporary societies is based on a complex combination of scientific, 
pedagogical and technical contributions, generally built from within the profession (Novoa, 2009). The 
teaching practice in Accounting involves the relationship between theory and practice, as well as the constant 
changes that affect the academy. It also involves the preparation for a variety of functions to be developed by 
their future professionals (Nossa, 1999; International Federation of Accountants, 2001). Such fact represents 
an important challenge to these professors (Miller and Becker, 2010). Therefore, they require a debate and 
actions regarding the education of professors involved in the Accounting Education process (Miller and 
Becker, 2010; Miranda el al., 2012), by the Higher Education Institutions of all countries. 

The Accounting professor must have technical and accounting knowledge, as well as some knowledge of 
related areas, teaching methodology, general culture and social aptitudes, in addition to the ability of constant 
integration of the various social, political, economic and legal phenomena in their pedagogical practice 
(Nossa, 1999). Such education is expected to mainly emphasize the professor’s reflection on his/her own 
practice (Kraemer, 2005; Ramalho et al., 2013). Laffin (2002) points out that didactic-pedagogical issues 
have not been considered as fundamental in the education of Accounting professors, by the post-graduation 
programs in Brazil. The pedagogical theoretical basis would help teachers to reflect on their daily practice. 
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Hernandes et al., (2006) reinforce the need of an education based on the reflection on the theory, so that 
accounting professors may encourage their students with critical thinking. Thus, according to Silva (2000) 
the Accounting professor needs to care for his/her teaching competency through updates and stricto sensu 
education (master’s degree and/or PhD), in order to develop the assimilation conditions of new management 
forms within the growing expansion of the professional practice. As a support, the recommendations for 
academic training of professors in the accounting area “requires a combination of original academic 
preparation ( stricto sensu education) in addition to follow-up activities that maintain or establish preparation 
for current teaching responsibilities " (Association to Advance Collegiate Schools of Business, 2010, p. 43). 
However, a weak academic education at stricto sensu post-graduation level of professionals in the area is 
observed in the Brazilian labor market ( Conselho Federal de Contabilidade, 2013), and such fact may cause 
an effect on the qualification of the contingent professors of Accounting undergraduate courses in Brazil. 

In this sense, what are the necessary qualifications for the teaching practice in Accounting undergraduate 
courses? The objective of this study is to investigate the perception of professors of the professional 
education area about the necessary qualifications for the teaching practice in Accounting undergraduate 
courses in Brazil. 

Swain and Stout (2001) surveyed graduates from 73 accounting doctoral programs in the United States. 
The researchers evaluated the individual development needs regarding the teaching process. The respondents 
pointed out having received minor pedagogical training during the doctoral program and that they had 
developed them mainly through self-training (Swain and Stout, 2000). 

Another studies (Slomski, 2008; Miranda et al., 2012; Vendruscolo and Behar, 2014; Vendruscolo and 
Bercht, 2015 ) surveyed the pedagogical practice of professors in Accounting undergraduate courses offered 
by Brazilian Higher Education Institutions. The results revealed in the study indicate they arise from the 
“initial and continuous education of professors, curriculum and school socialization, knowledge of the 
courses to be taught, professional experience, personal and professional culture, peer learning etc." (Slomski, 
2008, p. 3). "Teaching requires professional education for its practice: specific knowledge to appropriately 
practice it or, at least, the acquisition of knowledge and skills related to the teaching activity in order to 
improve its quality " (Slomski, 2008, p. 3). Thus, the teaching knowledge have a temporal and social 
character and leads to acquired and perfected processes within the practice and teaching career in the 
Accounting area (Slomski, 2008). 

Miranda et al. (2012) observed the qualifications of Accounting professors in three teaching education 
cores: academic qualification, professional qualification and pedagogical qualification. The professional 
qualification is related to the current accounting practices in the professional and pedagogical practice, to the 
systemic preparation for the teaching practice, whose knowledge arising from such qualifications pointed out 
in the study were: mastering the content, experience knowledge and didactic knowledge, respectively. The 
qualification component factors ascertained by the author are described in Table 1. 


Table 1. Accounting teaching qualification component factors 


Professor’s 

Qualification 

Component Factors 

Academic 

qualification 

The PhD degree (preparation for research), master’s degree, specialist, acting as referees or 
reviewers of scientific journals, exclusively acting at Higher Education Institutions, having papers 
published in scientific journals, participating in research associations or bodies. Thus, it was 
considered that the academic qualification should concern the teaching preparation for the practice 
of research on the issues he/she teaches. 

Professional 

qualification 

The accounting professional practice; as a consultant, advisor or referee; participation in 
professional associations or regulatory agencies; development of applied research involving the 
academy and the community; and the participation in projects of university extension programs 
with the involvement of the community. 

Pedagogical 

qualification 

Offer of teaching training programs; support to the teaching participation in research projects and 
scientific events; support to the teaching participation in stricto sensu courses offered by other 
Higher Education Institutions; promotion of scientific events contemplating teaching; have a 
stricto sensu, lato sensu and extension pedagogical education courses; teaching-related research 
projects; and teaching experience. 

Source: Miranda et al. (2012). 
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Vendruscolo and Behar (2014) investigated the professors’ competencies elements (knowledge, skills and 
attitudes) in Accounting undergraduate course. The authors have not identified a consensus on the literature 
about the pedagogical competencies of professors. The studies about the education of Accounting professors 
are recent and were tangent to the teaching competencies issue. They have not mapped them or identified 
their elements, not even approached how to develop them. However, in general, the presence of the three 
constitutive elements of competencies is observed: knowledge, skills and attitudes (Vendruscolo and Behar, 
2014). 

Vendruscolo and Bercht (2015) have analyzed the perception of affective aspects in their pedagogical 
practices, with 96 accounting undergraduate professors in Southern and Southeastern Brazil. The study data 
evidenced that Accounting professors do not adopt pedagogical practices through Accounting Information 
Technology, compatible with the distance education, revealing the training need in information technology. 

Thus, the contribution of this study is to theoretically discuss the education of Accounting professors, 
with empirical data, in order to produce more concrete answers, contributing to the mediation between the 
new base of the social reality and the requirements of specialized professionals to act in the organizations, in 
order to meet the evolutive challenges of society. 


2. METHODOLOGICAL PROCEDURES 

The research is applied to the higher education in Accounting. Based on the approach, the research may be 
classified as qualitative, since it represents an interpretative analysis of the collected empirical data and with 
quantitative elements for the descriptive analysis of data (Gil, 2007; Creswell, 2010). In relation to the 
objectives, it is descriptive. The technical procedure adopted was the survey through the online questionnaire 
via Google Forms, characterized by the direct questioning of the subjects whose behavior was expected to 
know and determine the characteristics and opinions of the study population (Gil, 2007 ). 

The respondents were 378 Brazilian professors, selected using non-probabilistic sampling through the 
typicality criterion: being professor in Accounting undergraduate courses and teaching professional education 
courses within the course’s program. The collection instrument contemplated the objectives of the research, 
the voluntary participation statement and the authorization for disclosure of the research anonymous data. In 
order to guarantee anonymity along the research, no name or identifying data of the respondents were 
disclosed. The denomination P n was adopted, where "P" refers to the respondent professor and "n" 
corresponds to the sample order number in the data base. 

The data collecting instrument was organized in multiple choice questions, from the Likert scale (Likert, 
1975; Gunther, 2003), in order to capture the perception of the research respondents in relation to the 
qualifications of a professor regarding the accounting teaching function for professors in Accounting 
undergraduate courses. The scales were: Essential, Very Important, Important, Of little Importance and 
Irrelevant. 


3. RESULTS AND DISCUSSION 

The analysis processes and data interpretation obtained by the survey technical procedure are shown. 

3.1 Profile of the Research Respondents 

In all Brazilian regions, most professors are male (64% of the participants). In the Northeastern and Northern 
regions this participation was even more expressive, with 67.4% and 76.5%, respectively. Out of the total of 
respondents, 74.6% are under 50 years old and 25.4% are older than 50 years. The age group with the largest 
concentration in all regions is between 41 and 50 years. Approximately, 53% of the professors act at private 
Higher Education Institutions, compared to 44.7% of public Higher Education Institutions, while 2.1% act in 
both. The regions with the highest presence of professors acting at private Higher Education Institutions are 
the Southern (63.3%) and Southeastern (67.3%) regions. This situation is inverted in the Center -Western 
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(64,1%), Northeastern (83,7%) and Northern (58.8%) regions, where the public institutions are more 
representative in the sample. 

The predominant academic education (78.8%) of the professors who responded the survey was Bachelor 
of Accountancy Science, in all regions. The Northern region stands out (94.1%), followed by the 
Center-Western (90,5%) and Northern (88,4%) regions. Professors from related areas represent 17.2% of 
participants and Other Knowledge Areas, represent around 4%. This participation is more influenced by the 
Southern (16.9%) and Southeastern (24.5%) regions. It has also been observed that 4% of the participants 
have double education, generally in accounting-related areas. 

In relation to the qualification of the sampled professors, 24.3% have a PhD (8.2% with post-doctorate) 
and 51.9% are concentrated in the Master's Degree - of these, 18% are currently enrolled in PhD programs. 
Thus, 76.2% of participants have the minimum title required by the Brazilian Ministry of Education for 
professional activity in undergraduate courses. This characteristic was similar in all regions, with emphasis 
placed on the Center -Western region, with a higher participation (85.7%), and the Northeastern region, with a 
lower participation (69.8%). Among professors with the minimum qualification of PhD, 71% are from public 
Higher Education Institutions, 61% of them are men and 39% women. 

The teaching experience resulted more uniform among the intervals analyzed. Most professors (65.3%) in 
the sample had up to 15 years of teaching experience. The largest contingent was in the interval between 5 
and 10 years, with 23.8% of the total. The interval between 15 and 30 years contemplates 29.9% of the 
professors. The sample also contains professors (4.8%) with more than 30 years of teaching experience. 
Confirming the theoretical references, 78.6% of professors have professional experience, in addition to the 
teaching practice. However, 21.4% only work as professors, mainly represented by professors who work at 
public Higher Education Institutions (61.7%). The research of Conselho Federal do Rio Grande do Sul 
pointed that 4.8% of Accountants also work as professors ( Conselho Federal do Rio Grande do Sul, 2013). 

3.2 Data Analysis 

Professors were consulted about the importance of qualifications and practical knowledge, necessary for the 
teaching function in Accountancy. The result obtained is indicated in Table 1, by scales. 


Table 2. Perception of respondents about the Qualifications of Accounting professors 


Qualification and Knowledge 

Essential 

Very 

Important 

Important 

Of Little 
Importance 

Irrelevant 

Total 

fr 

% 

fr 

% 

fr 

% 

fr 

% 

fr 

% 

Fr 

Technical Qualification 

245 

65,3 

91 

24,3 

41 

10,9 

1 

0,3 

0 

0,0 

378 

Pedagogical Qualification 

136 

36,3 

153 

40,8 

83 

22,1 

5 

1,3 

1 

0,3 

378 

Academic Qualification - Masters 

106 

28,3 

161 

42,9 

89 

23,7 

17 

4,5 

5 

1,3 

378 

Academic Qualification - Doctorate 

53 

14,1 

138 

36,8 

142 

37,9 

33 

8,8 

12 

3,2 

378 

Practical knowledge of Professors 

159 

42,4 

158 

42,1 

57 

15,2 

3 

0,8 

1 

0,3 

378 

Practical knowledge of Accountants 

164 

43,7 

144 

38,4 

64 

17,1 

6 

1,6 

0 

0,0 

378 


Technical Qualification was indicated as Essential (65.3%) and Very Important (24.3%) in the perception 
of 336 professors, representing 88.9% of participants. Practical knowledge as professors was considered by 
83.8%, as Essential (43.7%) and Very Important (38.4%) and as Accountants by 81.4%. The empirical data 
show that the sample teachers attach greater importance to the technical qualification and professional 
experience if compared to the others. Such importance can carry a greater weight in their continuing 
education, instead of the pedagogical and technological skills required in the teaching profession in the 
digital age. The market has required increasingly technically qualified professionals, but also shows a 
diversification of skills developed in their academic studies. In this regard, extensive training, covering the 
development of teaching skills result in a more significant contribution. 

The consideration of one of the respondents to the research illustrates the point of view of professors in 
relation to the importance of technical qualification and knowledge of the specific area: 
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P52 "For the good teaching of the accounting practice, the technical disciplines must be taught by 
professionals who work in the area. As we no longer have technical accounting courses, the Higher 
Education Institutions must prioritize the technical education (Accounting, Fiscal, Asset and HR 
bookkeeping) until the 5th period and professional disciplines should be taught from the 6th period on. The 
disciplines of accounting practice should be taught in computing laboratory with specific software". 

On the other hand, Pedagogical Qualification was considered Essential (36.3%) and Very Important 
(40.8%), by 289 respondents (76.5%) and Important by another 22.1%, totalizing 98.4% of participants. Data 
evidence that professors also consider pedagogical qualification relevant to the teaching practice in 
Accounting Science. It is observed with even greater significance given the technical knowledge that the 
consulted teachers also recognize the importance of Pedagogical Qualification. This involves knowledge, 
skills and specific attitudes, related to each other that enable teachers in conducting the process of teaching 
and learning accounting and enable a reflection on their practice. 

As stated in the literature review, the professor of Accounting courses did not receive academic training 
envisaging the needs for the teaching practice. Professors have been developing them in their own practice, 
which this study ratifies with empirical data. Professors have verbally expressed the comments indicated in 
Table 3. 


Table 3. Comments of respondents on the received academic training 

Professor 

Comment 

P186 

The undergraduate professor must share practice and teaching in order to develop competencies. 

P258 

The classroom is always a challenge, since in some classrooms a methodology works and in others the 
same methodology doesn’t, so professors should find these competencies and feel which methodologies 
should be applied in each classroom, discipline and each content. 

P363 

The professor, in his/her deepest essence, should be bound to the complex cognitive -pedagogical process 
of their students, searching to constantly improve his/her methods, in order to obtain a different, unique, 
process from his/her creativity, adapted to his/her personal dimensions. In other words, he/she should 
develop his/her own creative methodology, by applying a set of knowledge and experiences which expand 
as his/her teaching experiences accumulate. Experience improves it when he/she uses time as an extension 
process of the experiences he/she practiced. 


The comments of Accounting professors evidence the construction and development of their teaching 
competencies are attributed to the practice, trial and error, sharing of experiences with colleagues. Change in 
their methodologies through experience. They attribute to their teaching experience the constant 
improvement of their methods, without necessarily being aware of the process. 

By analyzing the comments, in the light of the Genetic Epistemology Theory (Piaget, 1975), they 
demonstrate the occurrence of a cognitive imbalance resulted by the “classroom” assimilation process. A 
beginning of awareness occurs by observing the need for change in action schemes (methodologies), as cited 
by P258 "in some a methodology works while in others the same methodology doesn’t". So, in an attempt to 
assimilate the object, the professor needs to modify their action (accommodation process): P258 "find these 
competencies and feel which methodologies should be applied in each classroom, discipline and each 
content". By doing this, professors develop cognitive structures in order to assimilate the object, developing 
thek teaching competencies, even if they are not completely aware of the process. 

In relation to the Academic Qualification for the function of Accounting professor, the Master’s degree 
received a higher weight (70.6%; 267 respondents) if compared to the Doctorate degree (50.5%; 191 
respondents), considering the Essential (28.3%) and Very Important (42.9%) scales. This result may be 
related to the qualification of the respondents, which indicated that 51.9% (196) have a maximum 
qualification of Master and 24.3% of PhD, or even to the parameters required by the Brazilian Ministry of 
Education. In this aspect, the voluntary comments of participant professors give an idea of the meaning 
attributed to the professor education, as shown in Table 4. 
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Table 4. Comments of respondents about academic qualification 


Professor 

Comment 

P148 

I believe higher education teaching lacks training of teaching techniques. We leave the Master’s and 
Doctorate degrees as researchers and have little or no teaching practice. Therefore, during the first 
teaching years we tend to reproduce the old board/marker system to pass on knowledge. I notice that even 
being aware of such change in the teaching techniques, only a few professors adopt or have autonomy to 
apply them. I think there is a lack of training, courses and sharing of experiences among professors. 

P94 

The institution where I work ignores the need for training professors for a better performance in their 
classroom. I need training and I see my colleagues need it too. But I believe this training should be partly 
optional, and partly obligatory and periodical. 

P220 

The Accounting Higher Education Institutions need to improve a lot with new Technologies and 
professionals with a good qualification in order to train competent professionals for the labor market. 


The feeling of deficiency in relation to the academic education received and the need for continuous 
training is expressed through the comments of professors in digital area. The concern of professors about the 
systematization of periodical training may also be observed, which could be provided by the Higher 
Education Institutions where they work. Professor PI 48 indicated that the academic qualification prepares the 
professor for research and not for the teaching practice and that professors adopt the modelling process for 
the conduction of their pedagogical practices. 

The results indicated the prevalence attributed by professors to the technical qualification and 
professional practical knowledges about the other teaching qualifications. However, they have attributed 
relevant weight to all of them, once almost all respondents scored up to the Important scale. This denotes a 
perception by professors of the complexity of the teaching practice in Accountancy. They also prove, with 
empirical data, what had been noted in previous studies, that is, that mastering contents by Accounting 
professors is not enough to the professional training in the accounting area. 


4. CONCLUSION 

The analysis results of empirical data contributed to reinforce the need for an academic education that 
envisages the practice requirements of professors and emphasize the need for a continuous education. 
Similarly, considering the social context of Accounting undergraduate professors’ practice, they revealed the 
feeling of deficiency in relation to the academic education received and the concern of professors about the 
systematization of periodical training which could be provided by the Higher Education Institutions where 
they work. Therefore, the Higher Education Institutions need to reflect on the pedagogical models adopted by 
stricto sensu Accounting courses in Brazil. 

The study indicated, also, that the professors adopt the modelling process for the conduction of their 
pedagogical practices. The academic qualification prepares the professor for research and not for the teaching. 
Mastering the contents is not enough for the teaching practice in Accounting courses. This denotes a 
perception by professors of the complexity of the teaching activity in Accountancy in digital area. 

Considering the respondents to the research are related to classroom courses and teach most disciplines 
only in this modality, we understand that further studies should be carried out with Accounting professors 
who teach in distance education. Specific research with respondents from distance education courses is 
recommended. As pointed out by Minayo (1994), in Social Sciences research, the social context must be 
considered due to the nature of this knowledge field, whose social education and configuration are specific. 

The results obtained in this paper may support institutional policies related to the continuous education of 
Accountancy undergraduate professors and the proposition of initiatives to develop them systematically. 
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ABSTRACT 

The twin forces of globalisation and internationalisation witnessed the global democratisation of higher education leading 
to the mushrooming of institutions of higher learning alongside with the rapid increase in student enrolments at all levels 
including postgraduate study. Despite the rapid developments in higher education, postgraduate study has been plagued 
with high attrition and low completion rates. Consequently, there is a need to look into delivery systems to ensure 
postgraduate supervisors use effective delivery systems in providing quality supervision. This paper investigated the 
supervisory and digital literacy practices among 40 supervisors in two public universities in Malaysia in the following 
four aspects: ethics, personal commitment, climate and use of digital tools. This mixed methods study collected data 
using questionnaires and semi-structured interviews. The findings revealed that the supervisors provided academic 
support, created a positive climate and guided their students about ethics in research to a great extent. The findings also 
revealed that supervisors should equip themselves with a diversity of supervisory practices, including making use of 
digital tools in managing their supervision practices more effectively. While a majority of the supervisors made use of 
digital tools in their supervision process, the interview sessions with six supervisors revealed that some of them reported 
being reluctant to make use of digital tools to check their students’ draft chapters. These findings have implications for 
the need of quality delivery systems especially institutional and academic support for postgraduate study. 


KEYWORDS 

Postgraduate supervision, digital literacy practices, supervisory practices, digital tools, academic support 


1. INTRODUCTION 

In the 21 st century, global experts acknowledge the rapid economic growth and development as well as 
impressive achievements made by higher education institutions in Asia. The cultural, educational and 
demographic depth of Asian civilisations is immense (Marginson, Kaur & Sawir, 2011). In line with such 
developments, empowering human capital through high levels of education is increasingly seen as being vital 
towards achieving better economic growth through the impact of research in driving innovation and 
competitiveness for any country in today’s competitive world, including Malaysia. Realizing the need of 
knowledge and innovation of becoming a developed country with higher income, the Malaysian government 
through the MinisUy of Higher Education (MOE) launched the National Higher Education SUategic Plan 
(NHESP) 2007-2020 on 27 August 2007. Under this plan, a special emphasis has been placed on providing 
better access to higher education, particularly in postgraduate programmes in the plan’s Phase 2: 
Strengthening and Enhancing (2011-2015) pillar. In this phase, a programme for financing postgraduate 
studies labelled ‘MyBrainl5’ has been included by the MOE and it aims to produce more doctoral graduates 
(PhD). MyBrainl5 was specifically established to develop a talent pool of high-level intellectuals to spur the 
growth of research and innovation in the counUy. Through MyBrainl5, the Malaysian government aims to 
produce a total of 60,000 Malaysian PhD holders by 2023. Up to June 2009, a total of 10,248 doctoral 
graduates have graduated from public universities, followed by 2116 graduates from private universities 
(MinisUy of Higher Education Malaysia, 201 1 ). 

Nevertheless, Sidhu (2010) stated that the number of doctoral graduates produced by each institution has 
been substantially low because close to 60% of them drop out at a global level. Various researches cited that 
one of the main reasons that conUibute to the atuition rate is postgraduate supervision (e.g. Moses, 1994; 
Wright and Lodwick, 1989). It takes an average 6-8 years to complete a PhD in most universities as a result 
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of inadequate facilities and resources as well as inadequate guidance from supervisors (Manyika & Szanton 
2001). For this reason, best supervisory practices were found to have a great impact on students’ success in 
completing their study and the supervisor became an important mechanism that ensures the students make 
good progress towards completion (Melissa Ng Abdullah & Evans, 2012). This calls for a need to re-examine 
the quality of postgraduate supervision in Malaysia. Though there are numerous studies that have looked into 
postgraduate supervision in the West there is little or scant empirical research on postgraduate supervision in 
Malaysia. Among the researchers that have explored this issue in Malaysia include Sidhu (2010) who 
conducted a comparative study that explored the perspectives of Malaysian and the UK supervisees on 
postgraduate supervision. Another study on postgraduate supervision was conducted by Norhasni Zainal 
Abidin (2006). Flowever, there is limited research that has explored this issue from the perspectives of 
supervisors in Malaysia. Therefore, this study aimed to investigate supervisors’ perspectives of postgraduate 
supervision. To be specific, it aimed to look into the supervisors’ perspectives on the supervisory and digital 
literacy practices in postgraduate supervision. 


2. LITERATURE REVIEW 

In today’s interconnected world, other than academic issues postgraduate supervisors are confronted with 
pertinent issues such as the criteria involved in being considered ‘literate’ in today’s digital age and the 
requirements involved in cultivating digital literacy for themselves as well as their supervisees. To a great 
extent, educators need to be aware that the use of digital literacies involves the shift from traditional 
print-based media to the current information and communication technologies that can be used in learning 
and teaching contexts (Lankshear & Knobel, 2008). In promoting learner-centred education, supervisors 
around the globe acknowledge that the focus of instruction has shifted to the learner and they need to make 
their learners cultivate cognitive and critical thinking abilities as well as research skills in their search for 
information and play more active and participatory roles in their postgraduate journey. There is a vast array 
of digital resources integrated as part of our social and work related practices that can cause inability of 
managing the choice of materials. Digital literacy involves the technical skills which will enable the effective 
usage of digital gadgets and it comprises executing tasks in digital environments such as constructing 
knowledge during surfing the web, deciphering user interfaces, searching in databases, creating and sharing 
content on the web, chatting in chat rooms and communicating in social networks. Kress (2010) 
demonstrated how even the traditional textbook has undergone significant changes in both appearance and 
content, becoming increasingly image-centred and moving away from the linear toward a more modular 
design framework. Similarly, Lotherington and Jenson (2011) advocate the use of digitally mediated 
communication in today’s postmodern world and call upon educators to put in place appropriate assessment 
practices that respect the varying academic competencies of all members of a learning context. For 
postgraduate students, living a literate life increasing means they have to learn how to navigate these spaces 
while managing one’s identify and online data and considering complex issues of privacy and representation 
(Buck, 2012). 

High quality supervision has been proven as an important element that facilitates students in fulfilling 
their academic potential (Cryer, 2000; Li & Seale, 2007). In view of that, Norhasni Zainal Abidin (2006) 
emphasized that best practices in supervision is a combination of factors which include the following: ( 1 ) the 
skill and commitment of supervisors (2) the ability and commitment of the students and (3) the existence of a 
policy or clear guidelines from institution. Thus, supervisors need to be clear about their roles and 
responsibilities in order to provide themselves with the necessary requirements and perform at their best 
during their supervision process. Since the quality of supervisory practices has a significant effect on 
postgraduate outcomes (Delany, 2009), it has become the interest of universities as well as governments to 
reliably improve the efficacy of postgraduate supervision. Van de Noort (2010) highlighted that supervisors 
should give guidance in terms of the planning of the research framework, helping to select relevant literature 
review and sources, conceptualizing research design and research methods as well as instrumental 
techniques, research data management and others. For example, some advice which should be given to 
students includes asking them to read books and journal articles, either in print or online form, that are related 
to the area of their investigation (Holdway et al., 1995). Additionally, during the period of writing-up of the 
thesis, supervisors are expected to keep in contact with students and respond to reasonable requests for 
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assistance (Abdulfattah Yaghi, 2006) and at the same time provide guidance on the writing and preparation 
of the thesis. This also includes commenting on at least one draft of the student’s thesis. Besides that, 
supervisors also need to respond through the drafts of students writing by giving constructive criticism of the 
draft chapter by chapter until the completion of the study. 

In addition, an effective supervisor must be competent, possess a solid research background and must be 
able to lead their students. Zhao (2003) highlighted that the main criteria in effective supervision is 
supervisory competence and emphasized that supervisors must be experts and be familiar with the students’ 
area of investigation. Spear (2000) claimed that postgraduate students should be able to be responsible for 
their studies as taking responsibility helps students to take charge of their postgraduate study. Hence, 
supervisors should treat the students as independent researchers who take initiative in proposing and 
executing research. McQueeney (1996) asserts that supervisors do not need to exercise too much control on 
their students’ progress as this can threaten the originality of the PhD and the autonomy of the novice 
researcher while too little control can delay and even lead to total failure. Thus, most of these researchers 
advocate the view that supervisors need to change their supervisory practices appropriately every time 
students move to another stage of academic writing. 


3. METHOD 

This study employed a descriptive research design with a mixed methods approach which involved both 
quantitative and qualitative data collection. The quantitative data were gathered from questionnaires while 
the qualitative data were obtained from interviews with the supervisors. A total of 40 postgraduate 
supervisors from two public universities in Malaysia (from three selected faculties comprising Education, 
Social Sciences and Science) responded to the questionnaires. These supervisors are referred to as Sample 
group A of the population. To maintain the respondents’ confidentiality, these respondents were coded as R1 
to R40. Sample group B of the study comprised six respondents (three from each university) who volunteered 
to take part in the interview sessions. Convenient sampling was used for the interview sessions. The 
supervisors were encouraged to give their honest views, opinions, comments, suggestions and 
recommendations on postgraduate supervision based on their experience as postgraduate supervisors. 
Keeping in line with research ethics, these supervisors were given pseudonyms such as Sl-S6.The 
questionnaire used in this study was adapted from a study conducted by Sidhu, Chan and Farhana Wan 
Yunus (2012). Apart from that, the researchers also adapted some items from the questionnaire developed by 
the AKEPT Centre for Teaching and Learning in Selangor, Malaysia. The questionnaire developed in this 
study was referred to as the Postgraduate Supervision Questionnaire (PGSQ hereafter). The PGSQ was 
divided into two sections. Part A examined the demographic profile of the respondents while Part B 
comprised 19 items regarding the supervisory and digital literacy practices of the supervisors. The ratings 
that were used ranged from 1 to 5 on a Likert scale (with the starting point of “1- To a very little extent” and 
the end point of “5- To a very great extent"). Table 1 below shows the four aspects in supervisory practices 
that were explored in this study. The reliability of respondents’ responses to the questionnaire was tested 
using the Cronbach’s coefficient alpha which was .895 (a=.895) indicating it was reliable (Cohen. Manion & 
Morrison, 201 1 ). 


Table 1. Distribution of survey items in Part B 


Aspects 

Number of Items 

Ethics 

5 

Personal commitments 

6 

Positive climate 

2 

Use of digital tools 

6 

Total 

19 
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4. RESULTS 

A total of 40 supervisors responded to the questionnaire in this study. Most of the respondents were female 
supervisors (62.5%) and the remaining were males (37.5%). In terms of their academic qualifications, a 
majority of them were PhD holders (67.5%) while the others possessed a Master Degree (32.5%). 

4.1 Supervisory and Digital Literacy Practices 

This study investigated supervisors’ perceptions of their supervisory and digital literacy practices in 
postgraduate supervision. The scope of practices encompassed the following four aspects: (1) ethics, (2) 
personal commitments, (3) climate and (4) use of digital tools. 

4.1.1 Supervisory Practices Based on Ethics 

Table 2 shows the findings of supervisors’ perceptions of their supervisory practices with regards to ethics. In 
general, it can be deduced that a majority of the supervisors demonstrated ‘to a great extent’ their supervisory 


practices during the supervision process (M=4.18, SD=.594). 

Table 2. Supervisors’ Perceptions with regards to Ethics (n=40) 

Ethics Mean SD 

Make students aware of issues in plagiarism 4.30 .687 

Ensure students adhere to ethical data collection and data analysis procedures 4.30 .608 

Clear about your roles and responsibilities as a supervisor in supervision 4.25 .670 

Ensure students are aware of the ethical codes for postgraduate study 3.92 .764 

Read the institution’s Supervisory Guidelines /Code of Practices for Supervision 3.70 .823 

Overall 4T8 2594 

Scale: 1= To a very little extent, 2= To a little extent, 3= To some extent, 4= To a great extent, 

5= To a very great extent 
SD : Standard Deviation 


From the results shown in Table 2, it was found that a majority of the supervisors demonstrated that they 
went to a great length to make their students aware of issues in plagiarism (M=4.30, SD = .687). Besides that, 
they also demonstrated (to a great extent) that they made sure their students adhered to ethical data collection 
and data analysis procedures (M=4.30, SD = .608) and it can be deduced that they were clear about their 
roles and responsibilities as a supervisor during the supervision process (M=4.25, SD=.670). On the other 
hand, supervisors ‘almost agree’ that they ensure their students are made aware of the ethical codes for 
postgraduate study (M=3.92, SD=.823) and assist their students to read the institution's Supervisory 
Guidelines /Code of Practices for supervision (M=3.70, SD =.823). 

Based on the interview findings, all six supervisors claimed that they demonstrated ‘to a great extent’ that 
they made sure their students adhered to research ethics. Female supervisor S2 pointed out that she warned 
her students from the first meetings with regards to the issue of ethics in research: 

“During my first meeting with my students I will make sure that he or she reads the 
institution's Supervisory Guidelines /Code of Practices for Supervision” 

Likewise, male supervisor S 1 also further stated that 

“I always ask for their soft copy to be checked for originality before checking students work 
further; in this way, I feel that my students will follow ethical procedures when doing their 
study ” 

4.1.2 Supervisory Practices Based on Personal Commitments 

Table 3 presents the findings of supervisors’ perceptions of their supervisory practices with regards to 
personal commitments. The results show that a majority of the supervisors demonstrated ‘to a great extent’ 
that their supervisory practices greatly assisted their students during the supervision process (M=4.30, 
SD=.648). 
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Table 3. Supervisors' Perceptions with regards to Personal Commitments (n= 40) 


Personal Commitments Mean SD 

Ensure students produce high quality work 4.50 .555 

Read and edit given drafts before consultations 4.35 .770 

Ensure students possess adequate research skills 4.35 .622 

Ensure students’ plans are divided into clear steps which lead towards completion 4.27 .716 

Provide prompt high quality constructive feedback 4.13 .723 

Ensure students know how to publish and where to publish 3.80 1.067 

Overall 430 .648 

Scale: 1= To a very little extent, 2= To a little extent, 3= To some extent, 4= To a great extent, 

5= To a very great extent 
SD : Standard Deviation 


As seen from Table 3, a majority of the supervisors demonstrated ‘to a very great extent’ the need to 
ensure that their students produce high quality of work (M=4.50, SD=.555). Besides that, they also 
demonstrated ‘to a great extent’ the need to read and edit students’ drafts before consultations (M=4.35, 
SD=.770) as well as ensure their students possess adequate research skills (M=4.35, SD=.622). Furthermore, 
the supervisors also agreed that they have to ensure their students’ plans are divided into clear steps which 
lead towards completion of their study (M=4.27, SD=.716) and they also provided prompt high quality 
constructive feedback during the supervision process (M=4.25, SD=.670). On the other hand, the supervisors 
chose the option ‘to some extent’ in relation to the need to make sure their students know how to publish and 
where to publish (M=3.80, SD =1.067). 

From the quantitative data, it can be deduced that 90% of the supervisors highlighted the importance of 
providing constructive feedback (both verbal and written feedback and on-the-spot feedback) in order to 
enhance the quality of their supervision. These findings were collaborated with data from the interview 
sessions in which the female supervisor S3 stressed, 

“/« order to make sure my students produce a good study I usually help them select what 
kind of journal they need to find or help them in googling the article. I read the same article 
too so that I can help my students better. Then 1 will assist them pick up the main point and 
link the point with another relevant article. I will do it first and show it to my student so 
that they will have the right idea. Later, 1 will let them do the critical reading and try to 
conned the contents of several articles. I also give them feedback and tell them what they 
need to improve.” 

4.1.3 Supervisory Practices Based on Positive Climate 

With regards to positive climate. Table 4 displays the findings of supervisors’ perceptions of their 
supervisory practices. In general, most of the supervisors demonstrated ‘to some extent’ that their supervisory 
practices helped to create positive climate (M=3.85, SD=.700). 


Table 4. Supervisors' Perceptions with regards to Positive Climate (n= 40) 


Positive Climate 

Mean 

SD 

Possess good communication skills 

4.12 

.791 

Ensure students engage with academic research communities 

3.38 

.979 

Overall: 

3.85 

.700 


Scale: 1= To a very little extent, 2= To a little extent, 3= To some extent, 4= To a great extent, 


5= To a very great extent 
SD : Standard Deviation 

The findings indicate that a majority of the supervisors possessed good communication skills (M=4. 12, 
SD=.791) in creating a positive climate with their students. On the other hand, the supervisors demonstrated 
‘to some extent’ the need to ensure that their students engage with academic research communities (M=3.38, 
SD=.979). 
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4.1.4 Supervisory Practices Based on the Use of Digital Tools 

The following section presents the supervisors’ perceptions of their supervisory practices based on the use of 
digital tools. The level of the supervisors’ perceptions of their use of digital tools looked into aspects such as 
their access to digital tools, use of search engines and use of data analysis tools. The study also explored 
supervisors’ use of digital tools to communicate with students with regards to checking and providing 
feedback to student drafts online and engaging with students for online discussions/supervision. From the 
quantitative findings presented in Table 5 below, it can be seen that supervisors reported that they had good 
access to online services (M=4.28, SD=.887) and frequently used emails (M=4.45, SD=.632) to communicate 
with their supervisees. The findings show that many supervisors also frequently make use of search engines 
(M=4.26, SD=.808) to look for information related to their students’ research areas. However, the use of data 
analysis tools in the supervisory process is a less common practice among supervisors (M=3.97, SD=.067). In 
addition, the supervisors engaged less frequently in checking and providing feedback to student drafts online 
(M=3.23, SD=.597) and engaged to a lesser extent with their supervisees for online discussions (M=2.98, 
SD=.876). 


Table 5. Supervisors’ Perceptions Based on the Use of Digital Tools (n=40) 


Ethics 

Mean 

SD 

Access to online services 

4.28 

.887 

Use of search engines 

4.26 

.808 

Use of data analysis tools 

3.97 

.067 

Email communication with students 

4.45 

.632 

Checking & providing feedback to student drafts online 

3.23 

.597 

Engaging with students for online discussions / supervision 

2.98 

.876 

Overall 

4.18 

.594 


Scale: 1= To a very little extent, 2= To a little extent, 3= To some extent, 4= To a great extent. 


5= To a very great extent 
SD : Standard Deviation 

The extent of the use of digital tools in supervising postgraduate students was further investigated in the 
interviews with the six postgraduate supervisors. The supervisors stated that they used their institution’s 
available online reporting system to record and track their supervisees’ performance each semester. Only two 
of the supervisors expressed the view that they made use of digital tools to provide feedback to their 
supervisees on a regular basis. According to male supervisor SI, 

“I use technology for functional purposes, that is to access networked sources, media and 
digital gadgets for my own purposes and I encourage my supervisees to do the same. 

Similarly I ensure that my students are at ease with using a range of search engines, online 
sendees and data analysis tools as well as powerpoint and other types of digital tools. This 
is important for all of us in the academic community. I admit I don ’t engage much in online 
forum discussions with my supervisees, mainly due to time constraints” 

One female supervisor (S5 ) had this to say, 

“I acknowledge that using technology > can help me supervise better. But I feel 
more at ease with working on hard copies of my students ’ draft chapters instead of working 
on their softcopy version. I can proofread and correct their work faster in print form. 
Generally, I don ’t make use of digital tools in my supervision process; I email my students 
mainly for setting appointments but prefer to provide feedback orally or on their 
hardcopies ” 

Female supervisor S2 expressed the following view: 

“As a supervisor, I believe I possess functional knowledge of digital tools. To be honest, 

I’m still learning how to use the ‘Review’ tool in Microsoft Word. ...because I’m over 50, 
it’s taking me longer to get used to editing students’ work digitally. But 1 am familiar with 
using search engines to look for new information and I read a lot of articles online and I 
encourage my supervisees to do that as well. ..also to locate Phd theses index etc. ” 
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5. DISCUSSION AND CONCLUSION 

The findings of this study indicate clearly that supervisors should equip themselves with a diversity of 
supervisory practices, including making use of digital tools in managing their supervision practices. The 
results of the study indicate that some supervisors are reluctant to make use of digital tools to check their 
students’ draft chapters. The supervisors demonstrated their supervisory and digital literacy practices to a 
great extent and provided good support to their students. This implies that supervisors were clear about 
providing guidance to their students during the different stages of supervision. There were four aspects 
explored in supervisory practices namely ethics, personal commitment, positive climate and use of digital 
tools. In the aspect of ethics, supervisors reported that they tried to make their students aware about 
plagiarism and they took steps to ensure that their students adhered to ethical data collection and data 
analysis. Supervisors also took measures to ensure that their students are aware of relevant regulations and 
legal issues, not only limited to plagiarism but also in terms of copyright, data protection, health and safety 
and any other ethical issues that might arise in research. McQueeney (1996) supports the claim that 
supervisors also have to make sure that their students understand university policies in preparing academic 
project reports. 

With regards to personal commitments, supervisors always ensure that their students produce high quality 
work by giving feedback and discussing plan of study with their students as well as giving academic advice 
during the supervision process. Supervisors also act as evaluators, monitors and gatekeepers (Holdaway et 
al., 1995). Supervisors have legal responsibility and ethics to monitor the quality of care that is being 
delivered to their students. In order to assure quality of care and enhance the professional functioning of the 
students, the supervisor constantly monitors and provides feedback regarding their students' performance 
(Spear, 2000). In addition, supervisors also reported that they maintained good communication skills with 
their students with regards to building a positive climate. Moses (1994) and Zhao (2003) emphasized that 
good supervisors necessarily led to trust and respect based on academic standards by possessing counseling 
skills. Based on gender, male and female supervisors have significant differences regarding their supervisory 
practices in the personal commitment aspect. 

In terms of the supervisors’ digital literacy practices, the quantitative findings indicated that there was no 
significant difference between male and female supervisors, indicating that the supervisors were comfortable 
in utilizing technology to enhance their supervisory practices. The interview data further indicated that male 
supervisors appear to encourage their supervisees to make use of a range of digital tools in searching for 
relevant information for their research topics in comparison to female supervisors. While there are hurdles in 
using digital tools in postgraduate supervision, the findings of the study indicate that supervisors do make use 
of such tools ‘to a great extent’ and acknowledge that institutional infrastructure can be further improved to 
support the use of such technologies on a wider scale. 
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ABSTRACT 

The empowerment of human capital through higher education is a key element that contributes to the growth of national 
income. In line with this, Malaysia under the MyBrainl5 initiative aims to produce a total of 60,000 doctoral degree 
holders by 2023. However, high attrition and low completion rates among postgraduate students to date have hindered the 
noble aspiration. Among the factors that have contributed to this are postgraduate supervisory practices and student 
limitations in terms of knowledge, and 21 st century skills such as critical thinking, autonomy and lifelong learning. 
Therefore, this paper aims to investigate students’ perceptions of their level of dependence on supervisors in postgraduate 
study in relation to digital literacy and academic matters in terms of providing motivational support, writing a proposal, 
collecting and analysing data and writing the final report. This exploratory descriptive case study involved 132 
postgraduate students from the largest public university in Malaysia. Data were collected using a mixed -methods research 
design through the use of a questionnaire and semi -structured interviews. The findings show that students’ level of 
dependence on supervisors for academic matters such as the need for was much higher compared to dependence in using 
digital tools. In terms of digital literacy, students articulated dependence on supervisors in aspects such as data analysis, 
data visualization and data management tools. With regard to academic study, students’ dependence on supervisors was 
highest at the proposal stage but reduced at the data collection and data analysis stages. Nevertheless, their dependence 
again increased at the final writing stages. The findings of the study suggest that postgraduate students lack autonomy for 
lifelong learning and hence appropriate steps need to be taken to improve the quality of postgraduate supervision in the 
university. 


KEYWORDS 

Postgraduate students, digital learning, lifelong learning, dependence, autonomy, postgraduate supervision, student- 
supervisor relationship 


1. INTRODUCTION 

In today’s competitive global markets, the development of a critical mass of human capital through higher 
education has become a key element that can contribute to the growth of national income. For this reason, the 
Malaysian government initiated a special programme referred to as MyBrain 15 which hopes to produce a 
critical mass of 60,000 graduates with doctoral degrees by 2023. Though this move has contributed to the 
increasing number of postgraduate students in institutions of higher learning in Malaysia, graduation rates 
have been low due to high attrition and low completion rates. Researchers such as Sidhu et al., (2014) have 
attributed this to student limitations in terms of knowledge, skills, student autonomy, inappropriate 
supervision and research learning environments. 

Sidhu et al. (2013) in their comparative study between Malaysia and the UK found the supervisees in 
Malaysia looked for a ‘people’ oriented supervisor who was a motivator and confidence booster. These 
students were also more dependent on their supervisors and had higher expectations of their supervisors 
compared to postgraduate students in the UK. Cryer (2006) further highlighted that though new postgraduate 
students usually are more dependent on their supervisors, they need to learn how to wean off their 
dependence as they progress along their postgraduate journey. Developments in research by researchers such 
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as Stefano et al, (2004) has indicated that learner autonomy can be fostered via organizational support 
(providing students with decision making roles), procedural autonomy support (offering students choices in 
use of different media) and cognitive autonomy support (opportunities for students to self-evaluate work). 

Consequently, there is a critical need for postgraduate students to become independent and autonomous 
lifelong learners. Knowles (1990) noted that the ultimate aim of all education should be lifelong learning if 
we are ‘to avoid the catastrophe of human obsolescence’ (p. 135). The concept of autonomy, self-direction 
and lifelong learning which were rather contentious and hotly debated issues in the 1990s have gradually 
become accepted ideas in education. Today, almost all universities and educational organisations, including 
the Ministry of Higher Education in Malaysia, have lifelong learning as an important attribute for graduates. 
Hence, students in today’s technology driven environments need to be responsive to change, inquiring and 
reflective in practice, through information literacy and autonomous, self-managed learning so that they leave 
tertiary institutions as lifelong learners. 

Autonomy, on the other hand, refers to the “capacity for making informed decisions about one’s own 
learning” and this can be developed through ‘introspection, reflection and experimentation through learner 
training’ (Sinclair, 2009, p. 1). To this discourse, Sidhu (2009) adds that for students to become autonomous 
lifelong learners in today’s ever changing digital learning environments, they must first be equipped with the 
right tools and strategies so that they are capable and willing to plan, organise, monitor and evaluate their 
own learning. Researchers (Sinclair, 2009; Sidhu, 2009) note that for students to be autonomous learners, 
they must be active learners who are capable and willing to engage themselves in academically stimulating 
and challenging learning environments. Besides that, students must be willing to take responsibility for their 
own learning. Taking charge of one’s learning is of utmost importance in postgraduate study if students want 
to leave academia as lifelong learners. 

Keeping in line with the noble aspirations of becoming a developed nation by 2020, Malaysia launched 
the Malaysian Education Blueprint 2015-2025 (Higher Education) in 2015. This Blueprint outlines 10 shifts 
to spur continued excellence in the higher education where shift number nine (9) pushes for the need for 
globalized online digital learning and shift four (4) focuses on developing a nation of lifelong learners in a 
bid to develop holistic, entrepreneurial and balanced graduates to fulfil the needs of a high-income economy 
(Ministry of Education Malaysia, 2015). If the postgraduate students are to become the nation’s critical mass 
of knowledgeable lifelong learners, it is only pertinent that supervisors empower students and move away 
from patriarchy, control and dependence. 

For success in completing an academic research study at the postgraduate level, students must be well 
equipped with current 21 st century skills which include digital literacy. In today’s cyber 21 5,1 century, students 
need to be digitally literate to access opportunities to learn, live and work effectively. The ability to utilize 
digital tools has also been identified as one of eight key competences for lifelong learning, as it ensures 
active participation in society and economy (European Parliament and the Council, 2006). Consequently, in 
today’s world of learning, both academics and students alike need to be digitally literate and be aware of the 
vast array of digital tools available for teaching and learning. The digital natives of today’s learning 
environments need to be well equipped with technical skills for the use of digital gadgets and software to 
manage and construct knowledge whilst surfing the web and deciphering user interfaces (Hall, Nix & Baker, 
2013). Today, both students and supervisors can even share content on the web, chat in chat rooms and 
communicate via social networks during on-line supervisory sessions. As such digital literacy involves a 
range of abilities from basic computing skills to the creation of multimodal texts and tools that can help 
postgraduate students in managing information, organizing the literature review and analyzing and 
visualizing data collected for the preparation of their final research report. 

Beside having to cope with the use of digital tools, these students also need to gain independence in 
online academic matters such registering for courses, settling fees and communicating online with 
supervsiors and other administrative personnel. Cryer (2006) notes that even though new postgraduate 
students are usually more dependent on supervisors for support in both academic study and digital literacy, 
supervisors must learn to strike a balance between control and autonomy. They need to slowly dissuade their 
students off their dependence and guide them towards pathways of autonomy and independence. This is 
important because at the end of the day, students need to emerge as confident researchers in their field of 
expertise so that they can contribute to the critical mass of knowledge workers required by the country. Even 
though postgraduate supervision is growing steadily to be an explored field of interest, there still exists scant 
empirical evidence on postgraduate supervision from the perspectives of postgraduate students, particularly 
in terms of their level of autonomy. Therefore, this study hopes to examine students’ level of dependence on 
their supervisor in coping with academic matters and using digital tools effectively in their postgraduate 
study. 
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2. THE STUDY 

This study was conducted in one of the largest public university in Malaysia and employed a descriptive 
research design with a mixed-methods approach. Data were collected through a questionnaire and 
semi -structured interviews. The questionnaire was aimed to investigate students’ perceived level of support / 
dependence on their supervisors in terms of coping with academic matters and using digital tools. A total of 
132 postgraduate students responded to the questionnaire whilst eight respondents agreed to be interviewed. 
These 8 respondents were coded based on gender (4 males and 4 females - coded as M or F) and academic 
study (PhD and Masters coded as P and M). In this study, these eight respondents were referred to as follows 
Rl-FP ( R=Respondent 1, a Female PhD student), R2-MP, R3-FP. R4-FP. R5-MP, R6-FM, R7-FM and 
R8-MM 

The questionnaire used in this exploratory study was adapted from Sidhu et al. (2014) Postgraduate 
Supervision Questionnaire to suit the objective of this study. It consisted of two sections: Section A explored 
the demographic variables of the students (e.g. gender, mode of study and academic qualification) whilst 
Section B consisted of 18 bipolar constructs that explored students’ need for support / level of dependence 
on their supervisor in coping with academic matters and using digital tools. The bipolar constructs in the 
student-supervisor relationship for academic matters examined students’ perceptions of the support they 
needed based on three stages in their postgraduate study which would indirectly reflect postgraduate 
students’ level of readiness for autonomy and lifelong learning. The three stages of study explored in this 
study for academic matters were: 1 ) the proposal stage, 2) data collection and data analysis stages and 3 ) the 
final writing stage. The scale used is shown in Figure 1 . Appropriate support on the left of the scale reflects 
the level of students’ dependence on their supervisors. Meanwhile the autonomy generation (on the right side 
of the scale) indicates students’ level of independence. 
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(indicating students’ dependence) 
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(leading to students’ independence) 


Figure 1 . Scale for Student-Supervisor Dependency 

The reliability of respondents’ responses to the questionnaire was tested using the Cronbach’s coefficient 
alpha in a pilot study which involved approximately 52 students from another public university in Malaysia. 
The Cronbach’s alpha reliability indexes for different dimensions in the instrument namely overall support 
required by students in postgraduate study (.96), dependence on supervisors for using digital tools (.94), 
academic support required by students at different stages (.94), and dependence on supervisors at various 
stages of study (.95) were identified at high level. Overall, the average Cronbach’s alpha reliability for the 
questionnaire was identified at .95 (a=.95) indicating it was highly reliable (Cohen, Manion & Morrison, 
2007). 


3. RESULTS AND DISCUSSION 

A total of 132 postgraduate students from a public university in Malaysia responded to the questionnaire. The 
demographic profile revealed that a majority of the respondents were females (65.9%) and 34.1% of them 
were males. As for the mode of study, 81.8% were full-time students while 18.2% were part timers. Out of 
these 132 respondents 85.6% were currently pursuing their masters whilst the remaining 14.4% were 
pursuing their PhD. 

3.1 Overall Support Required By Students in Postgraduate Study 

Table 1 below shows the amount of academic support students needed in their postgraduate study. Based on 
the findings in this pilot study, it can be seen that a majority of the respondents (79.5%) reported that they 
needed moderate academic support from their supervisors, indicating a moderate dependence on supervisors. 
This reflects a moderate readiness towards autonomy and lifelong learning. Another 10.6% of respondents 
claimed that they needed little support indicating that approximately 14 students were rather independent and 
were moving towards the path of autonomy. Another 13 students (9.8%) indicated that they were highly 
dependent and required much support from their supervisors. 
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Table 1. Level of academic support required by students (n=132) 


Level of Support 

Frequency 

Percentage (%) 

Low (1 - 1.99) 

14 

10.6% 

Moderate (2 - 3.99) 

105 

79.5% 

High (4 -5) 

13 

9.8% 

Total 

132 

100.0% 


3.2 Students’ Level of Dependence: Using Digital Tools 


The following section presents the students’ perceptions of their dependence on supervisors in terms of using 
digital tools in their study. From the quantitative findings presented in Table 2, it can be seen that students 
articulated independence on a variety of aspects. The highest level of independence was recorded for the use 
of referencing software (M=4.58, SD=.715) followed by the use of online academic matters (M=4.56, 
SD=.864) and use of search engines for research purposes (M=4.23, SD=.762). On the other hand, the 
highest dependence on supervisors was for the use of data visualisation tools (M=2.03, SD=.740) followed 
by use of data analysis tools (M=2.43, SD=.659) and use of plagiarism software (M=3.05, SD=.766). In 
addition, students expressed moderate dependence in terms of using social media in research, managing 
literature review online tools and use of technology for online supervision. 

The extent of support required by students was further investigated in the interviews with eight 
postgraduate students. Findings indicated rather similar findings obtained from the questionnaire. Almost all 
students highlighted that they were not dependent on their supervisors when it came to online academic 
matters. For example. Respondent R3-FP stressed that “I do not need any help from my supervisor on online 
academic matters because everything is on the postgraduate website. I can register for a semester, take a 
semester off and even apply for a change of supervisor.” Similarly Respondent R7-FM expressed that the 
“ university website was very user- friendly ” and she experienced no problems accessing it even though she 
was not as techno-savvy as her younger course mates. Likewise, a majority of them also voiced agreement 
that they encounter few problems when using available digital tools for referencing and manthey had little 
problems with the use of available digital tools for referencing and managing information obtained from the 
web. Respondent Rl-FP pointed out that during the orientation programme they were taught how to use 
various search engines and referencing software to locate information. She added, “We were also introduced 
us to some data management tools and the university library staff were very helpful . . . some of them taught 
us how to reference materials and where we could go for further help . . .but I think 1 need help and 
expertise in managing and writing my literature review.” 

Table 2. Student dependence on supervisors for using digital tools (n=l 32) 


Items 

Mean 

SD 

Online academic matters 

4.56 

.864 

Using Online search engines - research 

4.23 

.762 

Managing lit review online tools 

3.23 

.643 

Data Analysis & Analytics tools 

2.43 

.659 

Using referencing software 

4.58 

.715 

Understanding data visualisation tools 

2.03 

.740 

Use of plagiarism check tools 

3.05 

.766 

Use of social media in research 

3.79 

.462 

Use of latest tools e.g. mobile apps, etc. 

4.05 

.783 

Use of technology /media for online supervision /chat 

3.65 

.875 


Scale: Appropriate Support 1 — 2 — 3 — 4 — 5 Autonomy Generation 
(indicating dependence) (leading to independence) 


To this Respondent R6-FM highlighted that . . . 
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“my supervisor has exposed me to a number of software available for managing the writing of 
literature review such as Mendeley. In fact my supervisor is very techno savy as she is into 
computing software and has even created her own software called LIRAS to help us manage our 
literature review .” 

A majority of these respondents stressed that their main limitation lay in analytics and how to use data 
analysis software to analyse data. Respondent R2-MP stressed that . . . 

“I am rather weak in numbers and I really learnt a lot from my supervisor. He is good in 
statistics and I think I need his support in helping me use analysis software such as SPSS to 
help me use SEM and Rasch modeling”. 

Respondent Rl-FP further added that, 

“I collected and presented a lot of tables in my thesis but I was not able to present it in an 
attractive manner. My supervisor, he exposed me to a number of 2D and 3D visualization tools 
like charts, matrix, scatter plot diagrams and multidimensional pie-charts and histograms. Now 
I can say I am happy that my research study now looks like a PhD study. I really appreciate his 
help in helping me visualize my data effectively. ” 

Interview sessions also investigated challenges students faced with regard to digital literacy. A majority 
expressed their limitation again in terms of keeping up-to-date with available and supportive software tools to 
help them manage and analyse data collected. A few other called for the need for more effective plagiarism 
checking tools and need to up-skill their knowledge in data analysis tools such as NVIVO and latest 
computing software such advance Microsoft word software for thesis writing and formatting. 


3.3 Students’ Level of Dependence: Academic Matters 


The students’ level of dependence on their respective supervisors was also examined in terms of academic 
matters such as communication, feedback and supervisory meetings at three main stages of study namely 
proposal, data collection and data analysis as well as writing stages. The details of the overall results were 
computed and reported in Table 3 which presents the students’ level of dependence in seven (7) academic 
matters at different stages in their postgraduate study. At the proposal stage, they felt they needed to have 
effective communication (M=2.80, SD=.232) with their supervisors and hence they were rather dependent on 
highly experienced supervisors (M=2.54, SD=.162) who were able to provide them with proper guidance 
(M=2.68, SD=.155). More importantly, they also articulated a need for having a close relationship with 
supervisors as they needed them to provide strong motivational support. Students also indicated their 
dependence on quality feedback (M=2.77, SD=.163), regular meetings (M=2.54, SD=.140) and appreciated 
the need for a highly structured programme at this stage of postgraduate study. 


Table 3. Students’ level of dependence on supervisors at various stages of study (n=132) 


STAGE OF STUDY 

Proposal 

Stage 


Data Collection 
& Analysis Stage 

Final Writing 
Stage 

Items 

Mean SD 

Mean 

SD 

Mean 

SD 

Close relationship 

2.84 

.165 

3.05 

.018 

2.97 

.145 

Efficient communication 

2.80 

.232 

3.00 

.152 

2.92 

.205 

Quality feedback 

2.77 

.163 

2.98 

.068 

2.87 

.201 

Motivational support 

2.40 

.159 

2.92 

1.032 

2.87 

.162 

Proper guidance 

2.68 

.155 

2.92 

.024 

2.84 

.111 

Regular meetings 

2.54 

.140 

3.05 

.018 

2.79 

.260 

Highly structured programme 

2.58 

.166 

2.88 

.070 

2.73 

.165 

Highly experienced supervision 

2.54 

.162 

2.82 

.032 

2.63 

.128 

Overall mean 

2.72 


2.95 


2.85 



Scale: Appropriate Support 1 — 2 — 3 — 4 — 5 Autonomy Generation 

(indicating dependence) (leading to independence) 
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This high support and dependence at the proposal stage was also recorded during the interview sessions. 
This indicated a low level of readiness for autonomy and lifelong learning on their own. They admitted that 
they were most dependent on supervisors as ‘postgraduate study was a new chapter’ in their lives and they 
‘looked forward to developing a close relationship’ with their supervisors (R7-FM). Furthermore, 
Respondent Rl-FP stated that she needed the most support during the proposal stage and therefore, she made 
it a point to having regular meetings with her supervisor. She further stated that through the frequent 
meetings, she was able to experience good communication with her supervisor. 

“I really needed a lot of help from my supervisor when I was at my proposal stage. We had 
meetings almost three times a week for two months. The feedback I received on my proposal 
from her was also good and easy for me to understand. I can say that we had good 
communication during that time and I was really dependent on my supervisor. ” (R3-FP). 

Likewise Respondent R2-MP articulated that he Tacked confidence’ at this stage and was hence ‘very 
dependent ‘on his supervisor for ‘identifying readings, providing regular feedback on the writing’, and giving 
him ‘support, motivation and even guidance on how to survive postgraduate study’. 

The findings in Table 4 further illustrate that there was a slight shift in the students’ level of dependence 
at the data collection and data analysis stages (M=2.95) when compared to the proposal stage (M=2.72). 
Hence, respondents indicated there was little need for regular meetings with the supervisors (M=3.05, 
SD=.018) and they felt they could work independently and there was a little need of a close relationship with 
their supervisors during this stages (M=3.05, SD=.018). They however, indicated a need for quality feedback 
(M=2.89, SD=.068) and strong motivational support (M=2.92, SD=1.032). The results also show that they 
were most dependent on their supervisors in receiving highly experienced supervision at the data collection 
and data analysis stages (M=2.82, SD=.032). 

This sentiment was also reflected in the interview sessions. Thus, the meetings with their supervisors 
decreased. They however pointed out that they still looked forward to support and motivation from their 
supervisors, particularly on finishing their study on time. For example. Respondent R4-FP stated that she did 
not meet her supervisor as frequently as when she was in the proposal stage and their relationship became 
rather distant. Nevertheless, her supervisor still provided support to her. 

" Right now I am at my data collection and data analysis stages. As I spent more time collecting 
data from respondents so I have less meetings with my supervisor compared to previous stage of 
writing the research proposal. I think that is why we are quite distant but I still receive support 
from her each time we meet up. ” (PPR2) 

During the final stage of writing, the results reveal that respondents shifted back to dependence on their 
supervisors (M=2.85, SD=.861). Consequently, they articulated a need for a highly structured programme 
and looked forward to more regular meetings (M=2.79, SD=.260) as they were at the tail end of their study. 
They were still dependent on their supervisors for guidance, motivational support and quality feedback. They 
also appreciated efficient communication and looked forward to having a close relationship with their 
supervisors. During the interview sessions, they stressed that since they were at the final stages of their study 
they were more dependent and looked forward to a close relationship with their supervisors compared to the 
other stages of their postgraduate study. Respondent, R2-MP, pointed out that he actually had more effective 
communication with his supervisor as he had more frequent meetings at the writing stage. 

“I am in my writing stage now. I can say that my relationship with my supervisor is better 
compared to the other stages. We also have more efficient and effective communication as we 
have regidar meetings ...I think I am still rather dependent on my supervisor ” (PMC1 ) 

In addition. Respondent R3-FP further highlighted that she was rather dependent on her supervisor’s 
feedback and felt that even though she had reached the end of her study she was ‘not able to wean off 
successfully’. This is what she had to add: 

“I am at towards the end of my study and 1 know I should be a confident and independent 
researcher but the truth is I am still rather dependent as I think I still need my supervisor’s 
approval. I still like her to check my work ... I do not think / have been able to wean off my 
supervisor successfully and perhaps that is why I have not been able to achieve the 
independence required of a PhD student. . .but I am trying slowly. ” 
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4. DISCUSSION AND CONCLUSION 

The findings of this study showed that in general, a majority of the respondents were rather independent of 
supervisor support in terms of using digital tools but were rather dependent on supervisors in academic 
matters. Students expressed little need for supervisor support in aspects such as online academic matters and 
use of search engines for research purposes but called for support in the use of data analysis, data 
visualisation and data management tools. This calls for a need to further enhance digital literacy among 
postgraduates so that they do not lag behind the ever-changing demands of work and life. According to 
Shopova (2014), digital competence is crucial for effective learning and the need for students to adapt to 
adapt to the dynamically changing labour market. 

With regard to academic study, student dependence on supervisors was rather high at all three stages of 
academic study, indicating a lack of autonomy among postgraduate students. These students were most 
dependent at the proposal stage as they needed guidance, motivation and quality feedback. This is congruent 
with findings from a study by Grant (2003 ) who highlighted that it is crucial for the level of relationship to be 
high at the beginning of the supervisory process where the production of a thesis is the key component of a 
supervisory practice. Goldberg, Dixon and Wolf (2012) further reiterated that good communication at the 
beginning of the relationship helps to build strong bonds. Moreover, good communication between students 
and their supervisor is necessary for effective supervision. The findings also revealed that students’ 
dependence weaned off a little at the data collection and data analysis stages because at this stage students 
have to carry out data collection by themselves. This was also supported by Yin (2014) who states that 
students should learn to be more independent when collecting data. This slight independence however, swung 
back to dependence at the final stages of writing. Findings from the interview sessions indicated that students 
again fell back because their supervisors were still viewed as experts and a majority had not successfully 
learnt to wean off from their supervisors to grow as successful researchers in their own fields of interest. 

These findings have implications for postgraduate supervision. There is a need for a more holistic 
approach to postgraduate supervision that takes into consideration the tenets laid down by educationists such 
as Maslow and Vygotsky alongside the development of learner autonomy among postgraduate students 
(Sidhu et al., 2015). More importantly, both supervisors and students need to understand the delicate balance 
between control and autonomy. Effective supervision requires supervisors to employ a more student-centred 
approach in today’s age of technological advances as educators who see themselves as know-all content 
experts are redundant in today’s learning spaces. Supervisors need to listen to their students’ voices, get them 
engaged in dialogue with their own learning environments and help provide the necessary ‘scaffolding’ to 
reach their respective zone of proximal development. Researchers like Delamont, Parry and Atkinson ( 1998) 
assert that too much control can not only limit the development of autonomy of the novice researcher but 
more importantly threaten the originality of research study. Therefore, supervisors need to create a healthy 
supervisory culture that can help their students towards the path of increasing independence and lifelong 
learning. On the other hand, students too need to attend training seminars where they can be helped to grow 
as independent and critical thinkers. Students need to be empowered with lifelong learning skills so that they 
can be emancipated and encouraged to question and work towards developing as independent researchers and 
confident members of the disciplinary community. 
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ABSTRACT 

Informal learning allows learners to be in charge of their own learning process instead of being a content consumer. 
Harnessing mobile technology in informal learning field could help learners in taking a learning opportunity whenever 
they need either individually or in a group. This paper presents a small-scale study to investigate how people may use 
mobile technology for learning purposes in cultural heritage contexts. A focus group approach was used to capture 
preliminary results of user requirements. Based on these results, a scenario-based method was used to reflect a tangible 
picture regarding how people interact with mobile services. This study serves as an initial step of the series of gathering 
user requirements in developing a mobile location-based learning service. 


KEYWORDS 

Mobile Technology. Cultural heritage. Informal learning. 


1. INTRODUCTION 

Harnessing mobile technology in informal learning (mobile learning or m-learning) provides opportunities 
for people to learn regardless of their location either individually or in a group (Sharpies, 2000). M-learning 
helps encourage people to undertake new experiences in life as well as promoting lifelong learning 
(Fallahkhair et al., 2004, Vavoula, 2003). In addition, m-learning offers people the opportunity to learn 
within different contexts thus enabling people to learn while they are doing their daily activities (Brown et 
al., 2010, Kukulska-Hulme and Traxler, 2005) which takes place throughout people’s experiences (Cohen. 
1993). Learning from experiences could be considered as informal learning in which enables learners to be in 
charge of their own learning process instead of being a content consumer. 

Engaging in aspects of cultural heritage may form a significant facet of the informal learning process. 
Since cultural heritage reflects the identity of societies (Vidal Gonzalez, 2008), it could be important for 
people to learn more about the historical information that relates to heritage sites. This may help people to 
appreciate their history, which could further promote a sense of loyalty and engagement (UNESCO, 2013). 

Additionally, visiting historical sites reinforces the revival of the glorious past that the communities have 
had during a particular age. This would help people to derive a power from that history and to be proud of 
belonging to that community, and also maintain a link between the present and the past, which would 
stimulate the perpetuation of culture (Du Cros, 2001, Caton and Santos, 2007). 

Learning about historical stories and events that have taken place in a certain space not only attaches 
people to their roots (Poria et al., 2006), but also evokes their emotion and identity towards societies that they 
belong to (Poria et al., 2004). In turn this may inspire them to give more to serve their communities and. 
contribute to community advancement. 

Since people learn by interacting with each other (Tseng and Chen, 2014), and mobile learning facilitate 
the interaction between people in different contexts (Lam et al., 2010), effective mobile services should be 
introduced to enhance this interaction. This aspect could successively support informal learning in cultural 
heritage contexts. This paper presents an empirical study of how people use mobile devices for learning 
purposes with respect to culture heritage sites. A focus group approach was used to capture initial user 
requirements. A Scenario-based method was used to envisage the interaction process between users and 
mobile services based on the results of this study. 


51 



ISBN: 978-989-8533-43-2 ©2015 


2. FOCUS GROUP 

The present section describes the focus group approach that was used in this study including: the overall 
methodology, the methods that used to collect and analyse data, and finally, reporting the results. 

2.1 Methodology 

A focus group discussion was used in this study. Six participants took part in this discussion; their age ranged 
from 28-50, three male and three female. All of them are familiar with mobile technology. Participants were 
recruited among the PhD student community. A Doodle notification was sent to them by email asking to state 
day/time that is suitable for them in order to organize the discussion meeting. A thematic analysis method 
was used to analyze the data (Braun and Clarke, 2006). The data was coded manually and also electronically 
using QSR Nvivo 10. 

2.2 Data Analysis 

The data was analysed by using a thematic analysis method which consists of six phases that were set out by 
Braun and Clarke (2006) :1) Getting familiar with the data. 2) Generating initial codes. 3) Searching for 
themes. 4) Reviewing themes. 5) Defining and naming themes 6) Producing the report. 

As noted by Braun and Clarke (2006), themes could be ‘data-driven’ which is emerged from data without 
any prior ideas about it, or theory-driven’, which means that data has been coded around some initial ideas 
from the literature. Whilst both sources were used in this study, the obtained themes are more 
‘theory-driven’ as the researchers have had some initial ideas regarding this research from the literature. 
However, a few themes have also emerged from the data. 

The data were coded manually and electronically. First, manually: paper, highlighter, coloured paper and 
pencil. Second, electronically, using computer software: (QSR Nvivo 10) (Bazeley and Jackson. 2013, 
Bazeley, 2009). The electronic version was used to increase the reliability of coding data by looking at the 
data from a different angle. Using Nvivo helped to take in-depth insight to the dataset. Its value is that it is 
more accurate than do it manually in terms of avoiding missing some potential themes that might be 
important, due to the easy-checking by using software rather than go through the transcription manually 
every time to check (Basit, 2003 ). Two lists of themes have been obtained from using manual and electronics 
versions of coding. A list of final themes emerged by reviewing and combining these two lists. The coding 
process was conducted in three cycles: ‘generating initial codes’, ‘searching for themes’, and ‘reviewing 
themes’. 

It is necessary to clarify that throughout this paper we intend to use ‘code’ for the extracting information 
in the first level, ‘category’ for descriptive level of coding, and ‘theme’ for a more abstract level (Bazeley, 
2009). The details of how each phase of the thematic method was carried out are as follows: 

1 ) Getting familiar with the data 

This phase involved transcribing the verbal data, reading and re-reading through the entire dataset and 
also noting down some initial ideas. An initial list of potential themes was obtained through this phase. 

2) Generating initial codes 

This stage included deconstructing information from its original dataset into initial codes. These codes 
were assigned clear labels to act as rules for inclusion. The details of coding data in both versions (manually 
and electronically) as follows: 

Manually: The interesting patterns that are repeated across the dataset was highlighted (patterns related: 
problems, services, motivation) and also new patterns have emerged such as ‘a learning concept’. Afterward 
initial codes were generated as illustrated in Table 1. 
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Table 1. An example of coding data manually 


.if I go there I want to take picture and keep memory, I want to see 
memories, I wanna write down. Take picture and save them (manage pic- 
tures), I want to take direction as well, and some times I've been lost anyway, 
I want to find the history of things not just brief, I wanna go in depth like 
Charles Dickens, so, so. Probably I wanna to have a coffee in place like 
Charles Dickens lounge, I want when I go back have a look at what I've 


1- Talking about 
features and ser- 
vices. 

2- Talking about 
information and 

facilities 


Electronically: the Open Coding stage was carried out using Nvivo. This stage included exUacting 
information from the original dataset into initial "non -hierarchical codes’. Information was extracted for each 
participant to define a node or nodes based on the answer. For example, if the participants wrote down online 
ticketing and online booking for using mobile devices in cultural heritage sites, the node could be ‘online 
services’. As we went through the dataset any similar patterns were assigned to the nodes already defined for 
those patterns. When a new piece of information emerged, a new node was defined to assign this new 
information to. For instance, ‘learners and visitors’ are different than ‘usage of mobile devices’, so, a new 
node was defined to assign it. 

3) Searching for themes 

This phase included re-ordering initial codes, re-labelling and merging similar codes in order to ensure that 
labels for inclusion accurately reflect the contents. 

The initial codes identified from the previous phase were grouped together in meaningful categories. The 
related codes were assigned to a certain category in order to identify themes. A mind map was used in the 
manual version of coding (see fig 1). Furthermore, in the electronic version; re-ordering and re-labeling of 
nodes was carried out to identify categories. 

4) Reviewing themes 

This stage included breaking down the categories that were obtained from the previous phase into 
sub-categories to offer a clear insight to the meaning of categories. 



Furthermore, it is offer an opportunity to reviewing potential themes that might emerge from the defined 
categories in the previous phase. 

This phase was carried out in the manual version using pieces of coloured papers. The pieces were stacked 
onto an A2 paper to form more specific categories (sub-categories). 

Similarly, in the Electronic version: nodes were re-constructed into sub-categories. 

5 ) Defining and naming themes 

This phase included consolidation of codes from the three cycles of coding data. 

Themes were defined and named according to the meaning of each group that emerged from the previous 
phase. 
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At the end of the Defining and Naming Themes phase, the results from electronic and manual versions of 
coding were reviewed. A list of themes has been produced based on combining the results from these two 
versions (detailed description in the next section). 

6) Producing the report 

Based on the analysis of the data, a report was produced to summarise the analysis phases as well as the 
results. 

2.3 Results 

This study revealed a set of broad themes, which serve as initial requirements of developing a mobile 
learning service. These themes include: 

Learners and Devices: Learners could be categorized as groups and individuals, and also could be 
categorized as adults (elderly and young people), and children with their parents, grandparents, and teachers 
in a school trip. Learners interact with the cultural heritage sites using mobile devices which include: mobile 
phones, tablets and wearable devices (e.g. google glasses). In terms of kinds of mobile applications (apps), 
participants reported that is important to consider all types of operating systems to meet the multi -variation 
interest of people regarding types of mobile devices, “. . .different people has different preference “...is it 
a cross platform app?, ...what kind of apps that people comfortable with?”. 

A common view amongst participants was regarding considering user profile. Participants stressed that it 
is important to consider user preferences, “. . .personalize your app to suit your convenience...”, which means 
designing a user model that includes all user’s preferences based on user’s interests (Cocea, 201 1). 

The Learning Notion: The discussion has revealed different opinions about what the notion of ‘learning’ 
means. One of them considers learning as the process of getting information through courses, so, mobile 
learning is the ability to access online courses through a mobile device such as MOOCs (Massive Open 
Online Courses). The rest were divided between considering learning as; firstly, the process of retrieving 
information and retaining it for using when needed, secondly, as any type of information that people obtain 
during their daily life (e.g. looking at trains or buses time). 

Motivation and Attitude: The participants had diverse reasons for visiting cultural heritage sites and 
also for using a mobile device whilst there. The main reasons that motivate people to visit cultural heritage 
sites are: 1 ) learning, 2) entertaining, 3) discovering other countries cultures. Some of participants stressed 
that visiting historical sites could play a significant role in terms of helping people to learn about the history; 
either for themselves or their children, “7 would like to take my children to historical site to help them learn 
from them... ". 

Moreover, some of participants pointed out that the nature of people, who are interested in those sites and 
they enjoy history, would drive them to visit those sites. Entertainment could be a reason for visiting cultural 
heritage sites. Curiosity in discovering cultures, either their own cultures or other society’s cultures, might 
influence people to visit heritage and historical sites, “...I might go to visit cultural heritage or historical 
sites if I am on holiday in another country ” , “ I would discover society’s cultures, so the best way is to visit 
cultural heritage and historical sites... ”. 

Additionally, there was a sense amongst participants that a mobile app that provides almost all services that 
visitors/learners could need in cultural heritage sites, would encourage people to use it would have all their 
needs providing in one app. 

Services and Features: The main debate in the focus group discussion was about some existing services 
that are being used by people that could be utilised for cultural heritage contexts; such as the ‘Google Now’ 
app that notifies people about aspects based on their interests (Google). Some features that they are hoping to 
find in a mobile app; such as connecting the app with the social media to share information, photos, etc., keep 
memories and manage pictures and diaries, “...7 go there, ..,/ want see memories, 7 wanna write down, take 
picture and save them... ”. 

Furthermore, provision of interesting services may motivate them to visit cultural heritage sites. For 
instance, providing some useful information about some interesting aspects or facilities that could bring 
people attention, “. . . I like Charles Dickens; probably I wanna to have coffee in place like Charles Dickens ’ 
lounge... ” 

In addition, participants claimed that it is important to personalize the app to meet user preferences, which 
could in turn, motivate people to use it, “. . .personalize your app to suit your convenience ...” , also, it would 
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be advantageous if the service can provide a story narrator, that might attract children which in turn 
encourages parents or grandparents to use it when taking children for a day out, “ ...they can listen to a story 
while they are visiting the site... ” or utilize a quiz, “...quizzes for example... ”, “ ... you can make [quizzes] in 
different level...” Finally, participants suggested providing a unique and international code to be recognized 
everywhere which will help the app to be for a global use. 

Information: Information plays a significant role of developing mobile learning services for cultural 
heritage contexts. The reliability and usefulness of the information could encourage people to use this app. 
The debate in the discussion was about how people can obtain right information in a right time. Participants 
stressed that the quality of the information and the way that could be obtained is very important in terms of: 
1 ) generating reliable information, in which should be generated immediately at a real time. Managing and 
maintain data in an efficient manner ,“...how many places you gonna generate this information for, is just 
England!?...”. 2) pulling information from the cloud, which is easier in terms of it doesn't need to be 
generated immediately, however, it needs to be checked in terms of authenticity, “...if you are using 
information from the cloud, you have to think about the authenticity . . .” . 

Moreover, participants suggested providing some useful information for instance, how many visitors are 
visiting the site in a particular day or time, or some information about transportation. Those type of 
information could be helpful in terms of avoiding a crowdie day or to know about the type of transportation 
that is available,". .. it can give you information like taxis, buses, it could be helpful or how far from the bus 
station...”. Participants mentioned that enabling user to review comments that was generated by other 
visitors might help people to have an initial idea regarding a historical site before visiting it. 

Usability, Acceptance and Usefulness: Participants highlighted some factors that may affect people’s 
acceptance of using a new technology such as ease of use as well as provision of useful features and 
information. Participants stressed that asking a lot of questions and providing many choices could make the 
app complicated and not easy to use, which may dissuade people from using the software,". .. [if the app is] 
more complicated, more interaction and more question you will lose number of users... ”. 

Additionally, participants suggested giving the users a choice to disable or enable some services (e.g. 
switch off the notification service). This could give them an opportunity to choose what they prefer to acquire 
in a certain time. That may motivate them to use the app in which they do not feel restricted by such an 
annoying app (e.g. annoying notification), "... make it easy when you can switch things off or not... ” this 
would provide users with multilevel of interaction and thus, cater for the diversity of interests leads to 
increased adoption. 

Challenges and interventions: Participants underlined some challenges regarding using a mobile device 
in cultural heritage sites such as; a poor network quality in some remote places. 

The small screen size of some devices, such as mobile phones, might be not comfortable for elderly 
people who have got sight problems so they might prefer to use a tablet, “...that is an implication for elder 
people who may be would find it difficult to look at a small screen. . .”. 

The scalability might cause a problem in terms of the amount of retrieval data. For instance videos and 
images got tend to take a large amount of space in a mobile device memory (Alkhafaji et al., 2014). 
Nevertheless, using cloud computing as storage for the historical information might be considered as a 
solution for this issue. 

Furthermore, participants pointed out that to enable people to become engaged with the app, a level of 
trust must be established,"... [people] may not feel comfortable with something knows where they are...”. 
Confidentiality is an important aspect in such an app; people might not like apps that ask for personal 
information. This might lead to issues about why the app asks all these questions or how it knows about a 
particular aspect, “...when google suddenly give you an advert about some stuff you’ve been looking at, you 
are thinking how it knows that, and you thinking am not sure I like this... 

In addition, participants, emphasized that there are some people, probably the old generation, do not feel 
comfortable with the new technologies and may find it not easy to use, which may affect their attitude 
towards using a mobile device, "... is just I personally wouldn ’t, because I don ’t have that sort of easy to use 
a mobile phone...”, “...there is a generation of people who like to have a physical book rather than an 
app...”. However, it would be more useful to provide some interesting services such as personalizing the app 
that could bring their interest and encourage them to use it at cultural heritage sites, “...is like a trigger that 
makes somebody who never use that kind of things go and use it... ”, “...I can remember saying I wouldn ’t 
never have touch screen phone. ..then few years later you get you can ’t imagine life without it... ”. 
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Finally, another interesting issue has been mentioned by some participants who reported that the weather 
could be considered as a problem in the UK, which may prevent people to use their mobile devices in 
outdoor settings. However, some participants stated that using a Bluetooth headphone set might solve this 
problem. 


3. DISCUSSION 

The result of the focus group discussion acts as an introductory stage of the gathering user requirements 
process. It gives an elementary idea about how people would use mobile technology in cultural heritage 
contexts. In the other words, it shows how people would like to interact with mobile services. In turn, it helps 
in shaping the interaction process between users and systems. The results, while preliminary, suggest some 
interesting aspects in terms of interaction design. 

One important aspect is considering user’s profile; people would like to personalize their mobile app 
based on their profile (Cocea, 2011). In addition, even though people prefer the easiest and quickest ways for 
doing activities and interact with systems, they also like obtaining them in reliable and effective ways. With 
respect to using a mobile device to learn from cultural heritage sites, people prefer a mobile app that is easy 
to use and at the same time can provide rich and reliable information. 

Although, some people are not keen to visit cultural heritage sites, they like to do it for their children's 
sake or for getting extra services while visiting those sites (e.g. visiting Christmas shops in the Dockyard). 
Hence, despite the diversity of participants’ points of view, all of these opinions were revealed that learning 
about history and cultures would leads them to visit cultural heritage sites. 

Another important aspect that was highlighted in this discussion is collaborative learning. The result 
illuminates that most of people like to share knowledge and gain information via interacting with others. In 
other word, they like to do activities in a group which enables them to communicate with each other, and at 
the same time to share information, which may enhance their learning experiences (McLoughlin and Lee, 
2008, Laurillard, 2009). 

One more interesting aspect has emerged from data, which is the learning perception and how people 
understand learning. Interestingly, the result shows that people might learn incidentally and informally. 
Nevertheless, most of the time is happening unconsciously in which they do not perceive that they are 
learning. Given that, learning could be defined as acquiring information throughout a lifetime either through 
educational systems, formal learning, or life experiences, informal and incidental learning (Marsick and 
Watkins, 2001, Ainsworth and Eaton, 2010). 

The results revealed that some considerations should be taken into account when designing a mobile app 
in terms of interaction design. Main considerations include: user profile, ease of use, and sharing information. 
The next section will illustrate some results of the focus group study that has been used to embody user’s 
characteristics and preferences using a scenario-based design method( Carroll, 1999, Fallahkhair et al., 2004). 


4. SCENARIO-BASED DESIGN 

A scenario-based design method has been widely used in the field of HCI (Carroll, 2000), this is due to its 
ability to envision the interaction between users and services (Rosson and Carroll. 2002). Furthermore, 
scenario-based method helps to identify the suitable context of use and the main actors which consider the 
key elements for gathering requirements (Carroll et ah, 1998). We have developed a novel scenario that 
responds to some results of the focus group study presented as follows: 

Scenario: Dana and Sam are parents of three children; Sarah is 6, Jannah is 8 and Tom is 10. They are 
keen to get their children to learn about culture and history. They do believe that the best way to do it is 
taking them to visit cultural heritage sites. However, they concerned about how to get them enjoy the trip and 
learn at the same time, especially that the kids are different in their preferences and how they like to learn. 
Sarah and Jannah like to listen to a story whereas Tom likes quizzes. Sam’s noticed that some of his 
colleagues use a mobile app when visiting cultural heritage sites. This app enables them to personalise it 
based on their preferences and also give them an opportunity to choose how they prefer information to be 
presented (learning preferences). Sam downloaded this app in his mobile phone as well as to his wife’s 
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mobile phone and also to the kids’ tablets. During a summer holiday, they took the kids to visit the Southsea 
Castle in Portsmouth. Sarah and Jannah chose listening to the historic information about this castle in a story 
form. Sarah chose to listen to the story as an audio cartoon film, while Jannah chose to listen to a story that 
was told by a narrator. Tom chose to get information by taking a quiz that asks him to find the tunnel for 
instance and tick the right choice about why this castle was built? At the same time the family created a 
network between them, which enabled the parents to keep their kids tracks, share information, and also to 
enjoy their trip together as a group. 


5. CONCLUSION AND FURTHER WORK 

A Small-Scale study has been presented in this paper. The study was conducted to capture an initial 
requirement for developing a mobile location-based learning service with respect to cultural heritage 
contexts. A focus group approach was used in this study. This study serves as a bedrock in the process of 
collecting user requirement in the user-centred design approach. 

Although the current study is based on a small sample of participants, the results suggested a set of 
interesting themes, which will be useful to be considered in designing interactions of mobile learning 
services. These themes include: learners and devices; the notion of learning, motivation and attitude; services 
and features; information, usability, acceptance and usefulness and. challenges and interventions. A 
scenario-based method was used in this paper to illuminate some of the results regarding the interaction 
design between users and mobile services. 

The study that has been presented in this paper forms the first stage of a research project. This research is 
intending to develop a mobile location -based learning service with respect to cultural heritage contexts. 
There are a number of areas that we envision to carry out further work: Firstly, to conduct further steps to 
fulfil the eliciting of users requirements which includes: a) distributing a questionnaire form to obtain a 
wide-scale of data, b) Conducting additional interviews to gain in-depth details regarding using mobile 
devices for learning purposes. We are also planning to develop a task model based on the results of our 
questionnaire and interview studies using the socio-cognitive engineering methodology (Sharpies et ah, 
2002). Finally, a prototype mobile app will be developed as a proof of concept based on the task model. 
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ABSTRACT 

The development of numerical abilities was examined in three groups of 5 year-olds: one including 13 children 
accomplishing a numerical training in pencil-and-paper format (EG1); another group including 21 children accomplished 
a homologous training in computerized format; the remaining 24 children were assigned to the control group (CG). The 
participants were assessed at three successive times (tO, tl and t2) with a battery of validated tests assessing numerical 
abilities and fluid intelligence. At times tl and t2 we found differences between experimental groups and CG, while the 
children’s abilities in the two experimental groups were similar. We underline the crucial role of pre -syntactical and 
counting dimensions, accounting for a distinction between the experimental groups and control. Results are discussed 
with reference to the relevance for training activities of the presentation format (pencil-and-paper versus 
computer-assisted). Pragmatical and practical implications are also considered. 


KEYWORDS 

Early numeracy learning; Psychoeducational Training; Preschool; Cognitive Empowerment; Educational technology 


1. INTRODUCTION 

Many researchers are looking for exploring the basic structures that contribute to mathematical success in 
order to understand mathematical abilities development and to implement effective teaching strategies (e.g., 
Torbeyns, Gilmore, & Verschaffel, 2015; De Smedt, Noel, Gilmore, & Ansari, 2013). These studies 
evidenced the full complexity of mathematical acquisition processes starting from the preschool years. In this 
regard a growing body of studies has reported that early numeracy is developed during childhood even before 
the onset of formal education and it is progressively enriched at school (e.g., Jordan et al., 2009). Indeed, 
numerical processing is based on the development of two non-verbal mechanisms, such as the pre-verbal 
capacity to visually encode and enumerate a limited number of information (i.e., maximum 3-4 items), the so 
called Object Tracking System (Mandler & Shebo, 1982), and the capacity to discriminate two sets of items 
as a function of the number of elements expressing the quantity of each set (Approximate Number System; 
Dehaene, 201 1). The latter capacity implies that when more than four items are presented and serial counting 
cannot be carried out, the capacity to visually enumerate the objects fails and counting responses become 
more fallacious. Overall, from a typically developmental perspective this impasse can be overcome only 
when young mathematics achievers learn a symbolic enumeration system and some counting skills, which 
allow those children to process the numerousness of unlimited sets of elements (e.g., Jordan et al., 2009). 

However, further research shows that the development of mathematical skills predicts the successive 
mathematical achievements at school (e.g., Morgan, Farkas, & Wu, 2009) and the development of the related 
emotions, such as math anxiety, that is, the feeling of fear and tension interfering with mathematics 
performance (e.g., Jansen et al., 2013). For instance, early numeracy learning is slower in children attending 
primary school that successively are identified as students with dyscalculia (e.g., Passolunghi, Vercelloni, & 
Schadee, 2007). For this reason, during recent years the possibility of empowering early numeracy learning 
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by structured early psycho-educational interventions received much attention. Indeed, researchers (e.g., 
Fuchs, 2009; Siegler & Ramani, 2009) argue that early psychoeducational programs promoting 
mathematics-related learning can be very useful both for typically developing children and those being 
unproficient mathematics achievers. The principle underpinning the early interventions for the empowerment 
of numeracy-related learning is that, from the one hand, they can strength the mathematics knowledge and 
skills of typically developed children and. from the other hand, they can provide an opportunity to improve 
mathematics skills of at-risk children, in order to reduce even the negative emotional impact of scholastic 
failure on them (e.g., Jansen et al., 2013; Siegler & Ramani, 2009). Some literature shows that trainings 
based on technology supports have a great importance in the development of many different kinds of learning 
abilities (Meneses et al., 2012). Most literature focuses the attention on the number line trainings, starting 
from the beneficial effect of paper-pencil tools on children’s numerical development and their mathematics 
achievement (Moeller, Fischer, Nuerk, & Cress, 2015). This study underlines how computer-supported 
methods may considerably increase both the motivational aspect but also training efficiency in a great 
number of ways, for example implementing numerical trainings in general, ensuring adaptivity, accessibility 
and interactivity of training approaches. Some literature asserts that computer-assisted instruction is more 
effective and beneficial than traditional instruction for educating students (Gunbas, 2015). 

Other studies state that the pencil-and-paper and psychoeducational programs have the same efficacy in 
the enhancement of cognitive functions in childhood (Penna, Stara, & Bonfiglio, 2002; Penna & Stara, 2010; 
Ramani, Siegler, & Hitti, 2012; Passolunghi & Costa, 2014). For instance, a series of follow-up studies 
reports that specific metacognitive and cognitive (e.g., visuo-spatial attention and working memory) 
psychoeducational trainings can be successfully used in formal education in order to promote the 
empowerment of specific cognitive processes underpinning learning in the classroom (e.g., Fastame & Callai, 
2015). Previous research shows a correlation between individual differences in people’s school math abilities 
and the accuracy with which they rapidly and nonverbally approximate how many items are in a scene 
(Libertus et al., 2011). A longitudinal study explores the importance of kindergarten measures of 
phonological awareness, working memory, and quantity-number competencies as predictors of mathematical 
school achievement in third graders (mean age 8,8 years) (Krajewski & Schneider, 2009). Moreover, a 
further body of studies shows the efficacy of pre-literacy psychoeducational interventions in improving early 
numeracy skills (e.g., classification, seriation, counting) of pre-schoolers (e.g., Agus et al., 2015; Cirino, 
2011; Clements & Sarama, 2007). For instance, Clements and Sarama (2007) showed the effectiveness of an 
early numeracy learning intervention for young children lasted 26 weeks and based on a number of tasks 
such as counting, doing number comparison, and number recognition. 

For what concerns the Italian scenario, during the last decade different psychoeducational trainings have 
been developed in order to empower numeracy knowledge and related skills. In this regard, very recently 
Agus et al. (2015) conducted a preliminary study combining two pencil-and-paper and computer-assisted 
psychoeducational interventions aimed at enriching numerical knowledge and visuo-spatial abilities in 
second graders. The authors found that the combination of both the pencil-and-paper and computer-assisted 
versions of visuo-spatial and mathematical interventions was more effective on the empowerment of 
numerical skills than the execution of single treatments. 

The aim of this study was to verify the impact of two pencil-and-paper and computerized-assisted 
trainings on numerical intelligence of five years old children attending the last year of kindergarten. 
Specifically, we were interested in evaluating the effects of the presentation modality of the interventions in 
relation to the different aspects of numerical abilities (related to lexical, semantic, counting and 
pre-syntactical areas) and fluid intelligence empowerment. 

Therefore, in agreement with the literature, we hypothesised that: 1) the children that carried out a 
computerized training display a greater enhancement in mathematical learning (Praet & Desoete, 2014) with 
respect to children lacking any form of specific psychoeducational training; 2) the participants that performed 
the training in both modalities could have an analogous improvement compared to that of the children 
carrying out only the computer-assisted tasks, because the effectiveness of both modalities might be related 
to the novelty of the resources/tasks used (Swanson & Hoskyn, 1998). 
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2. METHOD 


2.1 Participants 


Fifty-eight children, enrolled in the last year of kindergarten in several Italian schools, took part in the study 
after their parents had provided written informed consent for participation. This study is only a part of larger 
researches involving more than 300 children. The sample comprised typically developing children, showing 
no signs of cognitive or perceptual deficits. Four pupils were excluded from the starting sample because they 
presented some learning difficulties disabilities. The children assessed belonged to the middle socioeconomic 
status. The research was conducted on the base of the ethical requests defined by the Italian Association for 
Psychology. Table 1 summarizes the socio-demographic characteristics of participants; there were no 
differences in relation to the sex variable (x 2 = .557, df = 2, p = .757). 


Table 1. Descriptive statistics in each group 



Control group (C) 

Experimental groups 




Training in pencil-and- ^ . . 

r I raining in 

paper format 

EGl 

computerised form 
EG2 

N 

24 

13 

21 

Percentage of women 

41.7 

53.8 

42.9 

PRE-TEST Age Mean in months at pre-test 

64.33 

64.23 

64.62 

Age Standard deviation 

3.30 

3.29 

3.81 

Lowest age 

58 

59 

59 

Highest age 

70 

70 

71 


2.2 Materials 

Pupils were presented a standardized battery, constituted by the Raven’s Colored Progressive Matrices 
(CPM, Raven, 1958) and by the BIN (Numerical Intelligence Scale; Molin. Poli, & Lucangeli, 2007), in 
order to have respectively a measure of fluid intelligence and of their numerical knowledge. 

The CPM (Raven, 1958) are used also for children in kindergarten (Italian adaptation, Belacchi et al. 
2008). 

The BIN (Molin. Poli, & Lucangeli 2007) evaluates four areas of numerical skills: the lexical subscale 
(i.e., assessing the ability to read and write Arabic numbers, moreover the skill to join the number-word to 
the exact digit); the semantic subscale (i.e., evaluating the ability to associate numerical sizes, dots and 
Arabic digits); the pre-syntactical subscale (i.e., appraising the capability to link numbers to their number 
representation and to order several quantities); the counting subscale (i.e., assessing the ability to recite the 
number-words sequence forward and backward, as well as the knowledge of the order of Arabic digits from 
1 to 5). 

After the pre-test phase (tO), children in the first experimental group (EGl) underwent the training 
program “L’intelligenza numerica I” (Lucangeli, Poli, & Molin 2003) and ‘Sviluppare l’intelligenza 
numerica I” (Lucangeli, Poli, & Molin 2010) in pencil-and-paper format. The remaining children were 
assigned to the second experimental group (EG2) that followed the homologous activities in computerized 
format (“Sviluppare l’intelligenza numerica I” by Lucangeli, Poli, & Molin 2010). The children in the control 
group (CG) carried out the regular curricular activities proposed by their teachers. The psychoeducational 
interventions employed in the current study consist of activities following the same conceptual and 
theoretical structure, developed to enhance numerical abilities in pre-schoolers pupils. The tasks were settled 
to empower and to favour a gradual and playful approach to the numerical knowledge in children, preparing 
them for primary school. The training enhances multiple processes: it favours the use of the Arabian code 
until 10, the learning of number names, the automation of the number sequence, the estimation of the weight 
of numbers, the evaluation of space, size and objects and the introduction of ordinality. Specifically, the 
software is composed of a series of tasks which are presented along the North Pole by an interactive tutor; he 
explains the instructions, gives feedbacks about the correctness of the performance and encourages the user 
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in case of failure. The level of difficulty of those tasks might be adjusted in order to adapt them to the 
efficiency of the user. 

Similarly, the pencil-and-paper program was aimed at enriching numerical knowledge, following the 
same principles. The program includes different units and learning situations, characterised by new elements 
and increasing complexity, organised in a series of printable worksheets for the activities. The 
pencil-and-paper activities were conducted by trained researchers. Both the computer-assisted and 
pencil-and-paper programs include several exercises promoting the metacognitive awareness and control of 
the cognitive processes involved in the activities. 

2.3 Procedure 

At pre-test (tO), post-test (tl) and follow up (t2), children were tested individually in two sessions, lasting 
approximately each twenty minutes. Each test was presented following the instructions contained in the 
original manuals (Belacchi et al., 2008; Molin, Poli, & Lucangeli, 2007). After the pre-test phase, 13 children 
(6 males and 7 females; M age =64.23, SD =3.2 months) were randomly extracted from the original sample 
and were presented the training activities in pencil-and-paper format (EG1), selected from “L’intelligenza 
numerica I” (Lucangeli, Poli, & Molin 2003) and ‘Sviluppare Tintelligenza numerica I” (Lucangeli, Poli, & 
Molin 2010). At the same time, 21 children (12 males and 9 females, M age = 64.62, SD = 3.81 months) 
were assigned to the second experimental group (EG2) that accomplished the homologous training in 
computerized format (Lucangeli, Poli, & Molin 2010). The remaining 24 children (14 males and 10 females 
M age = 64.33, SD = 3.3 months) were assigned to the control group (CG) that carried out the regular 
curricular activities proposed by their teachers. The groups were matched for chronological age. At the 
present, it should be stressed that the children in the experimental groups attended different schools from 
those in the control group. However, their socio-cultural context was similar, because the schools were 
located in neighboring villages. 

In the first experimental group (EGl) the pupils engaged in the pencil-and-paper activities that were 
carried out collectively during 10 weekly sessions that lasted approximately one hour each. On the other 
hand, each child in the second experimental group (EG2) was individually trained by the homologous 
computer-assisted intervention, for 10 weekly sessions, lasting 30 minutes each. When the trainings were 
concluded (i.e., tl - post-test after 3 months of the same scholastic year) and also after 6 months from the 
post-test phase (i.e., t2 - follow-up at the June of the same scholastic year), two further abilities assessments 
of the sample were carried out. Specifically the post-test was applied in all groups that were presented the 
same battery of tests originally administered at the pre-test - tO; the follow-up valuation was realized in two 
experimental groups (EGl and EG2). 

2.4 Data Analysis 

In order to verify the hypotheses of research, the potential differences between groups at pre-test in each 
dimension were assessed. For this purpose, a series of univariate Analyses of Variance (ANOVAs) were 
carried out on the scores calculated at pre-test, using the training group as factor (CG, EGl, EG2). All 
variables achieved the rules of the ANOVA application. These analyses did not revealed any significant 
statistical difference between the experimental groups and the control one in the CPM test [F=. 914; df=2;55; 
p= .407], BIN Lexical [F=. 217; df=2;55; p= .806], BIN Semantic [F=. 137; df=2;55; p= .872], BIN Counting 
[F=2.308; df=2;55; p= .109] and BIN Pre-Syntactical [F=. 544; df=2;55; p= .583]. 

On the bases of these outcomes, it was settled to investigate the information related to the effect of 
trainings in the groups at post-test and follow up (Keppel, 1991). Then the specific parametric statistical 
analyses were applied in order to verify the hypotheses, controlling the effect of age. Firstly the linear 
correlations were evaluated between all pairs of variables (at tO, tl and t2) by the application of Pearson’s r 
Coefficient. Formerly, to assess the specific role of trainings in the change of performance from pre-test to 
post test (in each dimension evaluated), a Multivariate Mixed design Analysis of Covariance was computed, 
where the repeated measures were constituted by the performances in two times (tO pre-test - tl post-test), the 
between factor was the “training group” (control group, experimental pencil-and-paper, experimental 
computerized) and the covariate was the age at pre-test. The application of this statistics was suitable in order 
to compare the performances in all scales for all groups in two moments, controlling the effects of 
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influencing factors. Consequently, another Multivariate Mixed design Analysis of Covariance was performed 
in order to evaluate if there was a retaining of achievements from the post-test (tl) until the follow up 
assessment (t2); then this analysis focused on the specific performances of experimental groups. 
Subsequently, a Discriminant Function Analysis was applied in order to identify the variables more useful in 
making the distinction among children in CG, EG1 and EG2. This analysis allowed to select the dimensions 
that better predict the probability of a child to belong to one specific group, useful to interpret group 
differences. 


3. RESULTS 

The first Multivariate Mixed design Analysis of Covariance was applied on all dimensions taken in account 
(CPM and BIN scales); the repeated measures were the pre-test and post-test assessments (tO and tl), the 
between factor was the training group (CG, EG1, EG2) and the covariate was the age at pre-test (Table 2). A 
significant effect of the repeated measures was found [F= 5.521, df= 1;54, p=.022, q 2 = .093] at tO and tl, 
highlighting that in post-test condition there are generally scores higher than at the pre-test. Moreover, it was 
also observed a significant interaction effect between the repeated measures (i.e., tO vs. tl) and training 
groups (i.e., CG vs., EG1 and EG2) [F= 4.787, df= 2;54, p=.012, q 2 = .151], that is, at post-test the scores of 
EG1 and EG2 are significantly higher than those of CG (Table 2). 


Table 2. Results of Multivariate Mixed Design Ancova comparing pre-test and post-test in groups CG, EG1, EG2 


Source 

Wilks’ Lambda 

df 

F 

P 

Eta 2 

Bonferroni’s Adjusted 
Group comparisons 

tO-tl 

.907* 

1 

5.521 

.0224 

.093 

to <tl 

tO-tl * age 

.938 

1 

3.570 

.064 

.062 

ns 

tO-tl * training 

.849* 

2 

4.787 

.012+ 

.151 

at 1 1 CG < EG 1 and CG < EG2 

scale 

.977 

4 

.260 

.903 

.005 

ns 

scale*age 

.965 

4 

.317 

.803 

.006 

ns 

tO-tl * scale 

.933 

4 

.715 

.583 

.013 

ns 

tO-tl * scale * age 

.945 

4 

.584 

.674 

.011 

ns 

tO-tl * scale * training 

.865 

8 

.844 

.565 

.030 

ns 


Note: + + p<.01 + p<.05 ns= not significant 


In order to focus on the effects of trainings, another Multivariate Mixed Design Ancova was applied, 
using the scores at post-test and follow up (tl and t2) as repeated measures. The training groups (EGl and 
EG2) as between factor and the age at pre-test as covariate. This analysis did not show any significant effect, 
emphasizing the preservation of cognitive achievement reached by the trainings, six months after the end of 
programs in paper-and-pencil and computerised formats (Table 3). 

Then, a Discriminant Function Analysis was performed in order to identify the variables discriminating 
the participants across the groups. We evaluated the issues related to the sample size, multivariate normality 
and multicollinearity (Huberty, 2005). In particular we met the criterion requiring that the size of the minutest 
group must be above the number of variables used as predictors. The discriminant function analysis was 
applied (Stepwise method) using CPM score and BIN subscales. The Box’s M statistic was not significant (p 
=.719), guaranteeing that the homogeneity of variance assumption was satisfied. At the last analysis step, the 
scales comprised in the unique significant function were the post-test BIN Pre-syntactical and post-test BIN 
Counting (Wilks’ X = .849, p<.0001), indicating that these two dimensions are discriminating the groups. 
Moreover, the discriminant function analysis had a reliable association with the training groups [x 2 =20.731, 
df=4, p < .001]. The function with the scales mentioned above was able to appropriately classify into groups 
totally 58.6% of participants [specifically the 70.8% of the CG (i.e. 17/24), the 38.5% of EGl (i.e. 5/13) and 
the 57.1% of EG2 (12/21)]. 
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Table 3. Results of Multivariate Mixed Design Ancova comparing post-test and follow up in the groups EG1, EG2 


Source 

Wilks’ 

Lambda 

df 

F 

P 

Eta 2 

Bonferroni’s Adjusted 
Group comparisons 

tl-t2 

.913 

1 

2.557 

.121 

.087 

ns 

tl-t2 * age 

.908 

1 

2.720 

.111 

.092 

ns 

tl-t2 * training 

.965 

1 

.971 

.333 

.035 

ns 

scale 

.872 

4 

.976 

.424 

.035 

ns 

scale*age 

.851 

4 

1.123 

.343 

.040 

ns 

tl-t2 * scale 

.835 

4 

1.105 

.355 

.039 

ns 

tl-t2 * scale * age 

.837 

4 

.987 

.410 

.035 

ns 

tl-t2 * scale * training 

.849 

4 

1.450 

.229 

.051 

ns 


Note: ns= not significant 


4. DISCUSSION AND CONCLUSIONS 

A wide literature states that psychoeducational trainings have some advantages and the same specific training 
can be settled and presented in two formats: computerized or pencil -and-paper (Penna, Stara, & Bonfiglio 
2002). In the pencil-and-paper format, for example, an important aspect can be the novelty, which could be 
identified in the presence of a new teacher, specialized in the promotion of training activities (Slavin, 2013; 
Agus et al., 2015). Educational technology is defined as a set of electronic tools and applications that help 
deliver learning materials and support learning processes to improve learning goals (Afshari et al., 2009, 
Meneses et al., 2012). Then technology can play a role in promoting an especially propulsive greater 
adaptability of the education system and in opening it to a range of values and attitudes as diversification of 
preparations, self-learning ability, aptitude confrontation and cooperation, ability to exploration and research 
(Vanderlinde, Aesaert, & Van Braak, 2014). 

This study shows the efficacy of both computer-assisted and pencil-and-paper trainings in enriching fluid 
and numerical intelligences. Indeed, differences between the experimental groups that use different training 
formats were not found. If there aren’t significative differences, the training with jcomputer technology opens 
up new directions for the successful training of numerical competencies that should be pursued as they may 
be particularly beneficial for those with special needs in numerical/mathematical learning. 

Overall, these outcomes, highlighted also by the discriminant analysis, underline the importance of the 
contents but not of the presentation modality in empowering early numeracy learning (Leh, 2011). Toll and 
Van Luit (2012) documented the validity of a psychoeducational intervention empowering reasoning, 
measuring, calculation, use of abstract symbols and number line skills that was developed for kindergarteners 
with mathematical problems. From an applied perspective, this is very crucial, because the benefits of early 
numeracy learning interventions in preschoolers at risk of developing mathematical learning difficulties can 
be already evident when those children attend the first grade of primary school (Dowker & Sigley, 2010). 
Since there is a high variability in pre-schoolers’ mathematics skills, during the last decades a series of 
trainings has been developed in order to enhance numerical knowledge in early childhood, that is, to promote 
early numeracy learning especially in low-performing learners (Riccomini & Smith, 201 1). Indeed, according 
to Clarke et at. (2006) the implementation of specific computer-assisted and/or pencil-and-paper 
interventions, mainly based on the use of explicit and simple instructions, on the employment of visual 
materials (e.g., cubes, number lines) and demonstrations about how to solve a problem, on breaking complex 
tasks in simpler units, seem to favour the empowerment of numeracy skills in early childhood, especially in 
kindergarten children at risk for severe mathematics problems. 

Another element of interest is given by the results of the discriminant analysis. This evaluation 
highlighted that the values of pre-syn tactical and counting scales are discriminant (Molin, Polin, & 
Lucangeli, 2007), distinguishing among experimental and control groups. Indeed, applying this function, the 
correct classification of group membership appears high in relation to CG, that is, in a condition in which 
children showed lower scores compared to those of the participants of the experimental groups. Also this fact 
confirmed the opportunity of differentiating between CG and experimental groups, but not in terms of the 
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modality (i.e., computer-assisted vs. pencil-and-paper) in which the psychoeducational intervention was 
proposed (Penna & Stara, 2010). In agreement with previous authors (Swanson & Hoskyn, 1998), the 
effectiveness of the training in both modalities might be related to the novelty of resources used and to the 
overall educational approach applied to elicit the empowerment of numeracy learning (Swanson & Hoskyn, 
1998). Furthermore, the efficacy of our psychoeducational intervention can be also related to the fact that it is 
aimed at empowering the metacognitive knowledge in numeracy learning. This is very relevant, because as 
In agreement with our results we can suppose that, as stated in the literature, what is important in order to 
assess the efficacy of an intervention is not the modality used, but the positive or negative effects on the child 
intelligence and learning (Penna & Stara, 2010). 

Concerning the limits of this work and the future developments, we think that it would be useful to 
deepen the analysis of the amount of achieved learning by using additional assessment tools. These tools 
should be appropriate to the recognition of the potential specificities linked to the two different training 
modalities (i.e., paper-and-pencil and computerized). 
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ABSTRACT 

This research aims to investigate the relationship among the awareness of self -regulated learning (SRL), procrastination, 
and learning behaviors in blended learning environment. One hundred seventy nine freshmen participated in this 
research, conducted in the blended learning style class using learning management system. Data collection was conducted 
in two ways; questionnaires for SRL scale “Motivated Strategies for Learning Questionnaire” (Pintrich and DeGroot, 
1990), and procrastination, and data log for learning behavior (report submission time). Students were asked to take the 
questionnaires in both pre and post class. As for learning behaviors, report submission time and one-minute paper 
submission time were collected using learning management system. The results revealed that internal value, 
self-regulation, and procrastination were fundamental elements that enhance the awareness of time management for 
learning plan, and positive time management awareness promoted to submit one -minute paper report within deadline, and 
regular report early. 


KEYWORDS 

Blended learning. Self-regulated learning. Procrastination 


1. INTRODUCTION 

Over 70 of the students in higher education postpone learning behaviors, until when they feel the necessary to 
do ( Schou wenburg. Lay, Pychyl, & Ferrari, 2004). This learning behavior is known as Procrastination. 
Procrastination was traditionally regarded as unsuitable learning behavior, in order to perform high learning 
outcomes. However, planned procrastination seems to be positive learning behaviors for high learning 
performance. In the case that learner who postpones learning acts for high learning performance, 
procrastination seems to be positive action, which can be regarded as high-level self-regulated learner. This 
research aims to investigate the relationship among the awareness of self-regulated learning, procrastination, 
and learning behavior in blended learning environment. 


2. PREVIOUS RESEARCH 


2.1 Self-Regulated Learning 

In order to foster autonomous learners, self-regulated learning is one of the important concepts for the design 
of learning environment. Many researchers conducted the research of self-regulated learning in experimental 
and practical educational settings. SRL is related to motivation, cognition, and self-control, as it is directed 
toward the accomplishment of learning purposes (Pintrich, 1999; Zimmerman, 1995). SRL learners are able 
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to apply self-control and self-evaluation (Deci et al., 1996). SRL is strongly concerned with metacognition, 
which leads to enhance the responsibility for learning on learner’s learning goal(Schunk and Zimmerman, 
1998; Zimmerman, 1986). Thus, the enhancement of the awareness of SRL can improve learning outcomes. 

Schunk and Zimmerman (1998) developed a three-phase model for SRL: forethought, 

performance/volitional, and self-reflection (see Fig. 1). Schunk and Zimmerman (1998) further compared the 
learning behaviors of novice and expert SRL learners in each SRL phase (see Table 1). In the forethought 
phase, skillful learners were able to articulate their final goal, and the necessary steps for its accomplishment. 
The features of both the goal and the steps toward it were constructive and clear. Skillful learners also tended 
to have internal motivation and high self-efficacy. In the performance/volitional phase, skillful learners try to 
maximize the effects of learning by monitoring the learning process; in the self-reflection phase, they sought 
to evaluate their learning performance independently and tended to attribute its quality to learning strategies 
and practice. Wolters et al (2003) also suggest the SRL framework, which contains the similar concepts 
concerned with controlling myself with cognitively in learning context. 




Performance or 
Volitional Control 




r 



A 

/ > 


Forethought 


< 1 

Self-Reflection 

* 

l 


N 


Figure 1. Three phase model of SRL (Schunk and Zimmerman, 1998) 


In e-learning context, high performer has time -management skills. Goda et al (2009) investigated the 
relationship between learning performance and learning habit. The results of their research show that high 
performer tends to make a regular life with time -management skills. Goda et al (2009) pointed out the 
importance of time-management skills in e-learning environments and time-management is concerned with 
all SRL phases, which Schunk and Zimmerman (1998) also indicated the similar points in face-to-face 
learning environments. 

Time management skill plays important roles in fruitful learning outcomes in both e-learning and 
face-to-face learning environments. Considering daily file, accomplishment of learning tasks following 
learning plan which learners made by them-selves indicates high SRL skill. In this sense, procrastination is 
not always harmful learning behavior for high learning performance. 


Table 1. Differences between naive and skillful self-regulated learners (Schunk and Zimmerman, 1998) 



Classes of self-regulated learners 

Self-regulatory 

Naive self-regulators 

Skillful self-regulators 

phases 

Forethought 

Nonspecific, distal goals 

Specific, hierarchical goals 


Performance goal orientation 

Learning goal orientation 


Low self-efficacy 

High self-efficacy 


Disinterested 

Intrinsically interested 

Performance/volit 

Unfocused plan 

Focused on performance 

ional control 

Self-handicapping strategies 

Self-instruction/imagery 


Outcome self-monitoring 

Process self-monitoring 

Self-reflection 

Avoid self-evaluation 

Seeking self-evaluation 


Ability attributions 

Strategy/practice attributions 


Negative self-reactions 

Positive self-reactions 


Nonadaptive 

Adaptive 
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2.2 Procrastination 

Skillful learners seem to use SRL skills appropriately, that is, SRL learners set appropriate learning goals 
including small steps, use the effective learning strategies such as monitoring strategy in performance phase, 
which leads to effective reflection for their learning. In this sense, procrastination can be one of effective 
learning strategies, in the case that learners can set appropriate learning time schedule, in order to accomplish 
the learning goals. Most research has found that procrastination has a negative effect on learning 
performance and can lead to physical and psychological problems (Hussain & Sultan, 2010). However, Chu 
and Choi (2005) suggested that some procrastinators performed high-quality learning outcomes with 
monitoring their learning behaviors, that is, procrastination is not always harmful for learning. Chu and Choi 
(2005) called them ‘active procrastinator’. Active procrastinator has several features to accomplish learning 
goal effectively and efficiently. Chu and Choi (2005) indicated four features of active procrastinator; 
outcome satisfaction, preference of time pressure, intentional decision to procrastination, and ability to meet 
deadline. Strunk, Cho, Steele, and Bridges (2013) proposed a 2 x 2 model of procrastination with two 
dimensions of time -related academic behavior and motivational orientation. 

Several research indicated that relationship between perceived sense of procrastination and 
procrastination behaviors. Most of the previous research indicated that negative correlation between 
procrastination behaviors and perceived sense of procrastination in traditional classfe.g., Tuckman, 1991; 
Howell et al, 2006). In e-learning, Goda et al (2014) reported that about 70 percent of learners were 
procrastinators, and the learning outcome (English test score) of procrastinator group was significantly lower 
than learning habit group. However, the relationship among SRL. procrastination, and learning behaviors is 
not clear. 

2.3 Research Purpose 

Reviewing sections 2.1 and 2.2, time management skill plays important roles in performing high quality 
learning outcomes, and procrastination under the learner’s control can be effective on the SRL, however, the 
overall relationship among SRL. procrastination, and learning behaviors is not clear. This study aims to 
investigate their relationship in blended learning environment. 


3. METHODS 


3.1 Subjects and Course 

One hundred eighty-three university students participated in this research. They all were the first-grade 
students, and took education introductory class. This course consists of fifteen classes. Main learning object 
is to understand the way to design the class with ICT. The students learned educational theories, principles, 
and history in the former eight classes. There were three criteria for grade; submission of one-minute paper 
after every class, regular reports (two times), and last report. Students should submit one-minute paper within 
a day for normal grade, but teacher accepted the submission one day delay (half a normal score was cut). 
One-minute paper must contain the class abstract and discussion. As for regular and last reports, teacher 
explained the report themes three weeks before the submission deadline. Students were required to submit 
one-minute paper and reports on LMS. 

3.2 Data Collection 

Students were asked to answer two questionnaires; Motivational Strategies for Learning 
Questionnaire(MSLQ) (Pintrich and DeGroot, 1990) and 2x2 time -related academic behavior scale (Strunk 
et al, 2013). The MSLQ, which consists of five factors (Self-Efficacy: SE, Internal Value: IV, Cognitive 
Strategies: CS, Self-Regulation: SR, Test Anxiety: TA; 44 items in sum, rated on a seven-point Likert scale; 
see Appendix A) was used for the subjective evaluation of learners’ SRL skill, but the factor ‘test anxiety’ 
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(four items) was eliminated, because this class did not use test for assessment. 2x2 model of time-related 
academic behavior scale consists of 22 items; seven items for Procrastination-approach, four items for 
Procrastination-avoidance, six items for Timely engagement-approach, and five items for Timely 
engagement-avoidance. Students were asked to rate each item in seven-point Likert scale (see Appendix B). 
The students were asked to complete the MSLQ and 2x2 model of time-related academic behaviors scale at 
the first class. The second method of data collection was a log that record the submission time of three 
reports, in order to collect the data for learning behaviors. Submission time was converted for analysis; 
earlier students submitted assignment, submission time increases, for example, when student submitted 
one-minute paper one hour before the deadline, submission time is one, in another case, when student 
submitted the regular report 100 hours before deadline, submission time is 100. About one-minute paper, we 
counted the late submission time through fifteen classes. 


4. RESULTS 


One hundred seventy-nine first grade students answered two questionnaires in a class. We conducted the path 
analysis, in order to investigate the relationship between SRL. procrastination, and learning behaviors. In 
section 4.1, we showed the descriptive data, the results of path analysis were shown in the section 4.2. 


4.1 Descriptive Data 


Table. 1 shows the average of each item. The score of each factor was calculated by the sum of each item in 
each factor. Average score of each factor and average time of submission are displayed in Tables 2, 3, and 4. 


Table 2. Average sum score of each factor in MSLQ 


Item Average score 

SD 

Self efficacy (min: 9 - max: 63) 

28.32 


8.23 

Internal value (min: 9 - max: 63) 

45.28 


8.08 

Cognitive strategy use (min: 13 - max 91 ) 

59.69 


8.73 

Self-regulation (min: 9 - max: 63) 

36.97 


4.61 

Table 3. Average sum score of each factor in 2x2 model of time-related academic behavior scale 

Items 

Average 

SD 


score 


Procrastination - Approach (min: 7 - max: 49) 


20.13 

6.93 

Procrastination - Avoidance (min: 4 - max: 28) 


14.07 

5.61 

Timely engagement - Approach (min: 6 - max: 42) 


24.72 

6.31 

Timely engagement - Avoidance (min: 5 - max: 35) 


21.32 

5.56 


Table 4. Average submission time (hour) and the late submission time of one-minute paper 


Items 

Average 

SD 

The late submission time of one-minute paper 

0.53 

0.81 

Submission time (hour) of regular report 1 

21.31 

32.18 

Submission time (hour) of regular report 2 

32.39 

43.70 

Submission time (hour) of last report 

39.55 

67.80 


These results revealed that there were big differences between individual students in items Self-efficacy 
in MSLQ, Procrastination-Approach, and all learning behaviors, due to big standard deviation. 
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4.2 Path Analysis 



Figure 2. Results of path analysis (Dotted line means negative relationship) 

Path analysis was employed using the average sum score and average submission time of three reports, 
and average late submission time of one-minute paper, in order to investigate the relationship among them. 
Figure. 1 shows the relationship among them using path analysis. Considering the statistic criteria of model 
fitness, this model is acceptable (chi2 = 51.862, df = 47, p = 0.290, RMSEA = 0.024, CFI = 0.992, TLI = 
0.989) 

The results indicated that internal value in MSLQ, and Procrastination - approach in 2 x 2 model were 
fundamental factors in order to raise the awareness of other MSLQ and time -related learning behaviors. In 
this overall model, negative and positive flow can be confirmed. In positive relationship, procrastination - 
avoidance affects on the late submission of one-minute paper negatively, and the late submission of 
one-minute paper also has negative relationship with the submission time of the first report. The submission 
time of the first report affects on both the submission times of the second and final report positively. In this 
relationship, learners who have high SRL awareness and active procrastination, keep the time for learning 
outcome submission. On the other hand, there seemed the learners who took negative time -related learning 
behaviors. Though learners had high SRL awareness, several learners have negative awareness of 
time-related learning behavior. Less procrastination - approach learners also have negative awareness of 
time-related learning behavior. However, the difference between positive and negative relationships is 
whether the late submission time of one-minute paper is mediated in the relationship. The learners who have 
high SRL awareness and time -related learning behavior seem to be aware of the engagement of weekly-task 
in every class. 
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5. CONCLUSION AND FUTURE WORKS 

This study aims to investigate the relationship between the awareness of SRL, procrastination, and learning 
behavior, in particular, the submission time of learning outcome, which is one of time -related learning 
behavior. The results revealed that SRL and procrastination positively affects on the learning behavior. Both 
positive and negative viewpoints of procrastination play important roles in the learning behavior; both the 
awareness of time -related learning behavior kept the deadline of the submission. Time management is one of 
important skills in SRL (e.g., Wolters et al, 2003; Barnard et al, 2009). The results of this study also indicated 
the awareness of that high SRL has positive relationship with the awareness of time engagement - approach, 
which is one of active procrastination elements. This results supports the findings of the previous research. 
However, not only active procrastination, but also passive procrastination keep the submission deadline, that 
is, even if learners regard time -related learning behavior as negative action, learners can do suitable learning 
behaviors. Waschle, Allgaier, Lanchner, Fink, and Nuckles (2014) show feedback loop between low 
self-efficacy and perception of goal achievement in procrastination in the investigation of the relationship 
between SRL and procrastination. But, in case that learners have high SRL awareness, learners can do 
appropriate learning behavior, even if learners have negative awareness of time -related learning behavior. 
Self-efficacy seems to play an important role in bridging learning behaviors and learning performance 
(Yamada et al, 2015). 

Future research should to clarify the relationship between SRL and learning performance. Schunk and 
Zimmerman (1998) indicated the differences between skillful and naive self-regulated learners. There seems 
to be differences between high and low learning performers in the relationship among SRL. procrastination, 
and learning behaviors. Attitude of ICT use in education should be considered in the future re-search. Usta 
(2011) indicated that negative attitude of ICT use has positive relation-ship with goal setting, time 
management, help-seeking, self-regulation, in particular, help-seeking behavior in e-learning setting can 
affect on learning performance (Goda et al, 2013). In this study, ICT was used in blended learning settings. 
Therefore, the effects of ICT use may affects on the results of this study. 


ACKNOWLEDGEMENT 

This research is supported by Grant-in- Aid for Scientific Research (B) No. 15H02935 


REFERENCES 

Barnard, L., Lan, W. Y., To, Y. M., Paton, V. O. & Lai, S-L. (2009) Measuring self-regulation in online and blended 
learning environments. Internet and Higher Education, Vol.12, pp.1-6. 

Chu, A.H.C. & Choi, J.N. (2005). Rethinking procrastination: Positive effects of “active” procrastination behavior on 
attitudes and performance. The Journal of Social Psychology, Vol.145, No. 3, pp.245-264. 

Deci, E. L„ Ryan, R. M. & Williams, G. (1996) Need satisfaction and the self-regulation of learning. Learning and 
Individual Differences, Vol.8, pp. 165-183. 

Goda, Y., Matsuda. T., Yamada, M., Saito, Y., Kato, H. & Miyagawa, H. (2009). Ingenious Attempts to Develop 
Self-Regulated Learning Strategies with e-Learning: Focusing on Time -Management Skill and Learning Habit. In T. 
Bastiaens et al. (Eds.), Proceedings of World Conference on E-Learning in Corporate , Government, Healthcare, and 
Higher Education 2009 (pp. 1265-1274). Chesapeake, VA: Association for the Advancement of Computing in 
Education (AACE). 

Goda, Y., Yamada. M., Matsuda, T., Kato, H. & Miyagawa, H. (2013). Effects of Help Seeking Target Types on 
Completion Rate and Satisfaction in E-Learning, Proceedings ofINTED 2013, pp. 1399-1403 

Goda, Y., Yamada, M„ Kato, H., Matsuda, T„ Saito, Y. & Miyamaga, H. (2015). Procrastination and other learning 
behavioral types in e-learning and their relationship with learning outcomes, Learning and Individual Differences, 
Vol.37, pp.72-80. 

Howell, A.J., Watson, D.C., Powell. R.A. & Buro, K. (2006). Academic procrastination: The pattern and correlated of 
behavioural postponement. Personality and Individual Differences, Vol.40, pp. 1519-1530. 


72 



12th International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2015) 


Elussain, I. & Sultan, S. (2010). Analysis of procrastination among university students. Proceclia Social and Behavioral 
Sciences, Vol.5, pp. 1897-190. 

Pintrich, P. R. (1999) The role of motivation in promoting and sustaining self-regulated learning. International Journal of 
Educational Research, Vol.31, pp.459-470. 

Schouwenburg, H. C., Lay, C., Pychyl, T. A. & Ferrari, J. R. (2004). Counseling the procrastinator in academic settings. 
Washington, D.C.: American Psychological Association. 

Schunk, D. El., and Zimmerman, B. J. (1998) Self-Regulated Learning: From Teaching to Self-Reflective Practice. New 
York, NY: The Guilford Press. 

Strunk, K.K., Cho, Y„ Steele.M.R. & Bridges, S.L. (2013). Development and validation of a 2 X 2 model of time-related 
academic behavior: Procrastination and timely engagement. Learning and Individual Differences, Vol.25, pp.35-44. 

Tuckman, B.W. (1991). The development and concurrent validity of the procrastination scale. Educational and 
Psychological Measurement, Vol.5, pp.473-470. 

Usta, E. (2011) The Examination of Online Self-Regulated Learning Skills in Web-based Learning Environments in 
Terms of Different Variables, The Turkish Online Journal of Educational Technology, Vol. 10, No. 3, pp.276-286. 

Waschle, K., Allgaier, A., Lachner, A., Fink, S. & Nuckles, M. (2014). Procrastination and self-efficacy: Tracing vicious 
and virtuous circles in self-regulated learning. Learning and Instruction, Vol. 29, pp. 103-1 14. 

Wolters, C. A., Pintrich, P. R. & Karabenick, S. A. (2003) Assessing Academic Self-Regulated Learning. Paper prepared 
for the Conference on Indicators of Positive Development: Definitions, Measures, and Prospective Validity. 
Sponsored by ChildTrends, National Institutes of Health. 

Yamada, M„ Yin, C., Shimada, A., Kojima, K., Okubo, F. & Ogata, H. (2015). Preliminary Research on Self-Regulated 
Learning and Learning Logs in a Ubiquitous Learning Environment, Proceedings of the 15 th IEEE International 
Conference on Advanced Learning Technologies, pp.93-95. 


Appendix A Motivational Strategies for Learning Questionnaire (Pintrich and DeGroot, 1990) 


Factor 

Self-Efficacy 


Intrinsic 

Value 


Test Anxiety 


Cognitive 
Strategy Use 


Item 

Compared with other students in this class, I expect to do well. 

I’m certain I can understand the ideas taught in this course. 

I expect to do very well in this class. 

Compared with others in this class, I think I' m a good student. 

I am sure I can do an excellent job on the problems and tasks assigned for this class. 

I think I will receive a good grade in this class. 

My study skills are excellent compared with those of other students in this class. 

Compared with other students in this class, I think I know a great deal about the subject. 

I know that I will be able to learn the material for this class. 

I prefer class work that is challenging so I can learn new things. 

It is important for me to learn what is being taught in this class. 

I like what I am learning in this class. 

I think I will be able to use what I learn in this class in other classes. 

I often choose paper topics I will learn something from even if they require more work. 

Even when I do poorly on a test, I try to learn from my mistakes. 

I think that what I am learning in this class is useful for me to know. 

I think that what we are learning in this class is interesting. 

Understanding this subject is important to me. 

I am so nervous during a test that I cannot remember facts I have learned. 

I have an uneasy, upset feeling when I take a test. 

I worry a great deal about tests. 

When I take a test, I think about how poorly I am doing. 

When I study for a test, I try to put together the information from class and from the book. 

When I do homework, I try to remember what the teacher said in class so I can answer the questions correctly . 
It is hard for me to decide what the main ideas are in what I read. (R) 

When I study, I put important ideas into my own words. 

I always try to understand what the teacher is saying even if it doesn't make sense. 

When I study for a test, I try to remember as many facts as I can. 

When studying, I copy my notes to help me remember material. 
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When I study for a test, I practice saying the important facts over and over to myself. 

I use what I have learned from old homework assignments and the textbook to do new assignments. 

When I am studying a topic, I try to make everything fit together. 

When I read material for this class, I say the words over and over to myself to help me remember. 

I outline the chapters in my book to help me study. 

When reading, I try to connect the things I am reading about with what I already know. 

Self- I ask myself questions to make sure I know the material I have been studying. 

Regulation When work is hard, I either give up or study only the easy parts. (R) 

I work on practice exercises and answer end of chapter questions even when I don’t have to. 

Even when the study materials are dull and uninteresting, I keep working until I finish. 

Before I begin studying, I think about the things I will need to do to learn. 

I often find that I have been reading for class but don’t know what it is all about. (R) 

I find that when the teacher is talking, I think of other things and don’t really listen to what is being said. (R) 
When I’m reading, I stop once in a while and go over what I have read. 

I work hard to get a good grade even when I don’t like a class. 


Appendix B: 2 x 2 measure of time -related academic behavior scale (Strunk et al, 2013) 


Factor 

Procrastination - 
Approach 


Procrastination - 
Avoidance 


Timely 
engagement - 
Approach 


Time 

engagement - 
Avoidance 


Item 

I more effectively utilize my time by postponing tasks 
I delay completing tasks to increase the quality of my work 
I put off starting tasks to increase my motivation 

I feel a stronger state of flow in my tasks when working closer to a deadline 
I intentionally wait until closer to the deadline to begin work to enhance my performance 
I delay tasks because I perform better when under more time pressure 
I rarely have difficulty completing quality work when starting a task close to the deadline 
I put off tasks for later because they are too difficult to complete 
I avoid starting and completing tasks 
I often delay starting tasks because I am afraid of failure 
I delay starting tasks because they are overwhelming 

I work further ahead of the deadline at a shower pace, because it helps me perform better 

I believe I can successfully complete most tasks because I start work immediately after being assigned a 

task 

I do my best work well ahead of the deadline 

I start working right away on a new task so that I can perform better on the task 
I complete my tasks prior to their deadline to help me be successful 
I begin working on difficult tasks early to achieve positive results 
I start my work early because my performance suffers when I have to rush through a task 
I do not start things at the last minute because I find it difficult to complete them on time 
I begin working on a newly assigned task right away to avoid failing behind 

When I receive a new assignment, I try to complete it ahead of the deadline to avoid feeling overwhelmed 
On extremely difficult tasks, I begin work even earlier so I can avoid the consequences of putting it off for 
later 
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SOUND AS AFFECTIVE DESIGN FEATURE IN 
MULTIMEDIA LEARNING - BENEFITS AND 
DRAWBACKS FROM A COGNITIVE LOAD THEORY 

PERSPECTIVE 


Anke Konigschulte 

University of Applied Sciences Bremerhaven, Germany 


ABSTRACT 

The study presented in this paper investigates the potential effects of including non-speech audio such as sound effects 
into multimedia-based instruction taking into account Sweller’s cognitive load theory (Sweller, 2005) and applied 
frameworks such as the cognitive theory of multimedia learning (Mayer, 2005) and the cognitive affective theory of 
learning with media (Moreno, 2006). Proceeding from the assumption that sound is an incisive means to affect people’s 
emotional state it is argued that sound may also be well suited to stimulate involvement and motivation in learning 
situations, thereby bringing the learner to invest more mental effort into learning, which finally leads to better learning 
performance. 

This paper refers to an experimental case study, which was carried out within the framework of a Master’s Thesis at the 
University of Applied Sciences Bremerhaven (Germany). In order to investigate the cognitive effects of including sound 
into multimedia learning, two groups of 1st semester Digital Media students were asked to learn about a historic subject 
using two different experimental designs: One version of a prototypical learning application consists of a photo slideshow 
with accompanying audio narration and another version consists of the same material supplemented with environmental 
sounds that illustrate the content of the lesson. 

Comparing both groups, the results don’t reveal significant differences in learning performance. However, the subjective 
mental effort ratings of the participants are identified as a positive predictor for the performance score and are thus 
hypothetically discussed as being an indicator for learner motivation. The analysis finally confirms that the learner 
involvement, which is a measure relating the performance score and the mental effort ratings (Paas et al., 2005), during 
the subsequent achievement test is significantly higher when sounds were presented during instruction. These results 
suggest that the inclusion of sound may have positive effects on motivation and learning, which is according to a 
cognitive -motivational theory. 


KEYWORDS 

Multimedia Learning, Instructional Design, Sound, Cognitive Load, Motivation, Involvement 


1. INTRODUCTION 

Through the introduction of digital audio in the early 1980s and the evolution of Internet capabilities, 
appropriate and cheaper software and (portable) audio players, multimedia developers are nowadays able to 
easily store, manipulate, (re-)use, and thus fully integrate sound in (instructional) software. However, even 
though there have been significant changes in technology since the heyday of radio in the 1930s, only “little 
has been done to advance the single notion of audio in instruction”. Holmes and LaBoone (2002) argue (p. 
57, cited in Schlosser & Burmeister, 2006). 

Research within the realm of auditory perception (e.g. McAdams & Bigand. 1993; Van Leeuwen. 1999) 
as well as common practice in film sound (e.g. Chion, 1994) and computer game sound (e.g. Jprgensen. 
2009) suggests that sound offers many promising possibilities in instructional design that remain largely 
unexplored. Without a doubt, auditory information participates fundamentally in the development of 
knowledge by facilitating the acquisition, processing, and retrieval of information in many ways ( McAdams 
& Bigand. 1993, p. 1-9; Gaver, 1993). However, the integration of non-speech audio, that is sound and music 
in its different forms, has been apparently neglected in the current state of research and practice in 
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multimedia learning. Although some applications indeed add sound (because it is possible), there are very 
few research studies that formally evaluate the effects of it, respectively, provide guidelines or a structured 
way to integrate sound in instructional media (Bishop et al., 2008). 

1.1 Objective and Framework 

Sound is a very different medium for representing information than visual images. Due to natural factors, 
such as the absence of some mechanism to shut the ears (“eyelids”), and the omnidirectionality of hearing, 
listening is much less limited than seeing. Therefore, people encounter sounds and noises every time and 
everywhere, so that they usually ‘fade them out’ as long as it is not relevant to (re-)act to them. In everyday 
listening, people naturally seek to denote sounds to their sources, actions, and events that cause them in order 
to get important details about the immediate surrounds; it is, like Smalley (1996, p.79) puts it, “a question of 
living and acting in the world, ultimately of survival”. When a person walks along a road at night hearing a 
car he/she most likely does not focus on the sound itself, e.g. its pitch and loudness, but compares the sound 
to his/her memories for the known objects that make that sound. With attending to the whole scene, drawing 
from existing schemas and experiences the person will understand the situation, e.g. it is a car with a strong 
engine approaching quickly from behind and I should step out of the path (Gaver, 1993, p. 1). It may be for 
these reasons that sound, even not alarming by itself such as an even faraway motor’s engine or flat tire’s 
faint thumping, can immediately activate existing schemas and is thus generally more effective than images 
for gaining attention (Posner et al., 1976; Bernstein & Edelstein, 1971, cited in Bishop & Cates, 2001, p.ll). 
Sounds, which also carry strong affective significance, may add yet another dimension to the audiovisual by 
making the environment more immersive, tangible and by provoking emotions, such as suspense. That way, 
even unintrusive sounds, as for example waves hitting the shore or whistling tree leaves, may also effectively 
hold attention over time and reduce distraction of competing stimuli (Thomas & Johnston, 1984, cited in 
Bishop & Cates, 2001, p. 12). Chion (1994) points out, that it is the sound’s physical nature as vibrant 
phenomenon and its omnipresence that interferes with and affects people’s perception more than the image, 
especially when people do not give conscious attention to it. Therefore, sound can be an insidious means of 
affective and semantic manipulation (pp.33). Sound facilitate a rather intuitive view of the information it 
presents than the visual. 

The present study draws mainly on the emotive qualities of sound and tries to approach in how far the 
inclusion of sound (i.e. non-speech audio) to audiovisual learning material can affect the learners’ affective 
state; in particular, if it can stimulate learners’ involvement and motivation to learn in a way that learning can 
be enhanced. 

Nevertheless, numerous of experimental media studies reveal that the actual challenge for the 
instructional multimedia design is for the processing demands to not exceed the limited capacity of the 
cognitive system, which otherwise easily end up in creating a cognitive overload and decreased learning 
performance (e.g. Moreno & Mayer, 2000; Mayer et al., 2001 ). Moreno and Mayer (2000, p. 1 18) accordingly 
conclude that any additional material “that is not necessary to make the lesson intelligible or that is not 
integrated with the rest of the materials will reduce effective working memory capacity and thereby interfere 
with the learning of the core material”. It is thus especially a cognitive perspective on learning that raises 
doubts about the effectiveness of including sound in instruction. Accordingly, Bishop and Cates (2001, p.6) 
argue that “without a strong theoretical cognitive foundation, the sounds used in instructional software may 
not only fail to enhance learning, they may actually detract from it“. In line with this, this work aims to 
integrate and design sound with regard to research -based theory of how students learn in order to 
successfully enhance learning. 

1.2 Theory and Predictions 

Cognitive theories of multimedia learning, such as Sweller’s cognitive load theory (Sweller, 1988; Sweller, 
2005), and Mayer’s cognitive theory of multimedia learning (Mayer, 2005), clarify memory processes of 
learning and problem solving and have thus contributed a solid foundation to guidelines that help 
practitioners to design multimedia instruction more effectively. According to the cognitive theory of 
multimedia learning (CTML, see Figure 1), different stages of memory process audio/verbal and 
visual/pictorial information in distinct, independent but cooperative channels. Meaningful learning only 
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occurs through active processing in working memory, that is, selecting, organizing and integrating 
information with the aim to construct coherent mental representations. These processes are partly guided by 
prior knowledge stored in long-term memory. 


Multimedia Sensory Long-Term 

Presentation Memory Working Memory Memory 



Figure 1. CTML, according to Mayer (2005) 

The working memory processes only a very limited amount of information at any one time, which is a 
major impediment when learners are required to learn new material. Therefore, learning instruction should 
avoid to provoke processing and storage that is not relevant for learning, which is the primary objective set 
by Sweller’s cognitive load theory. Cognitive load represents the load that performing a particular task 
imposes on the learner’s cognitive system and is determined by the interaction of environment, learner, and 
task characteristics. On this account, three types of cognitive load are distinguished (see Sweller et al., 1998, 
pp.259): The intrinsic load is the inherent difficulty level of the topic to be learned, extraneous load is 
considered as unnecessary processing and is determined by the manner in which information is presented, 
and germane load is devoted to learning, that is, processing, construction, and automation of schemas. 
Cognitive load theory and CTML focus on instructional methods aimed at reducing extraneous load and that 
way freeing memory capacity for an increase in germane learning activities (Sweller, 2006, p. 168). Several 
empirically validated methods for eliminating extraneous load via the instructional design have been 
published yet, such as the modality principle, redundancy principle, coherence principle, signaling, 
segmentation, and many more (e.g. Mayer, 2005; Mayer & Moreno, 2003). 

In Moreno and Mayer’s (2000) study investigating the inclusion of seductive details in multimedia 
instruction, participants perform significantly worse in subsequent retention and transfer tests when they have 
learned on basis of narrated animations with looped bland background music or with sound effects. With 
reference to CTML, the authors suggest that sound and music constitutes unnecessary (extraneous) load that 
might prime the activation of inappropriate prior knowledge as the organizing schema in working memory. 
Furthermore, additional sounds risk to reduce the learners’ capacity for integrating the relevant verbal and 
visual material to a coherent system, which impedes learning due to memory’s capacity limitations. 

Cognitive load theory and CTML, however, focus mainly on freeing working memory capacity by 
reducing extraneous load, which assumes that learners would automatically spend all their available 
resources in productive ways, that is, in germane learning activities. Accordingly, Paas et al. (2005, p.25) 
criticizes: “Until now, cognitive load theory (CLT) has focused on the alignment of instruction with 
cognitive processes, without recognizing the role of motivation in training”. It is important to note though, 
that learners’ investment of cognitive resources can be rather affected by relatively instable factors such as 
general orientation, motivation and state of arousal. The cognitive affective theory of learning with media 
(CATLM) extends CTML to better integrate the role of motivation, that is, affect and motivation that mediate 
learning by increasing or decreasing the amount of cognitive resources that learner invests on the task 
(Moreno, 2006; Moreno & Mayer, 2007). On that account, different germane load inducing techniques have 
been investigated in recent studies. These reveal that, for instance, emotional design features (e.g. Plass et al., 
2014), interesting additional material (e.g. Park et al., 2011) and methods such as imagination assignments 
(e.g. Cooper, et al., 2001) may lead to an increase in mental effort investment and thereby learning 
performance. 

These considerations form the theoretical case for including sound into multimedia instruction presuming 
that the addition of sound may be a useful tool to direct attention, add emotion and “playfulness” to the 
lesson and encourage imagination. According to CATLM, it is hypothesized that sound in multimedia 
lessons can stimulate learner's interest and enjoyment and achieve a greater level of motivation to engage in 
deeper learning. Sound as affective design feature thus encourages the learner to invest more mental effort 
into learning, which lead to a better learning performance. 
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2. EXPERIMENT 

The aim of the experiment is to investigate whether the addition of corresponding environmental sounds to a 
narrated photo slideshow increases the learner’s involvement and motivation, thereby encourages to invest 
more mental effort into learning, which, according to CATLM, leads to better learning performance. 

Two groups of students are asked to learn using two different experimental designs, which are with or 
without sound illustrations. Via subsequent achievement tests and questionnaires the performance score, 
invested mental effort, motivation, and user experience related data is measured and compared using 
quantitative methods. 

2.1 Learning Material 

For the experimental test, a learning application is prototypically implemented that teaches the user about the 
history and (re- (construction of the medieval cog found in Bremen, which is exhibited in the German 
Maritime Museum Bremerhaven. The learning application, which is installed on desktop computers in the 
computer lab of the University, is segmented into five screens, each one presents a slideshow lasting about 
three minutes including two or three pictures and accompanying audio explanations. The learner can use the 
standard media control buttons play/pause, rewind, repeat, volume and can move to the next or previous 
screen using the mouse when they want to. 

One version of the application includes environmental sounds, which adapt during runtime according to 
the content of the lesson. As the sounds are included with the aim to attain positive effects on learning 
efficiency, the requirements is set to minimize the (feared) likelihood of cognitive overload and distraction. 
Considering the theoretical considerations summarized above, it is important with the sounds to not just add 
“appealing” extraneous load. As auditory information can be far more attention-grabbing than visual pictures, 
only relevant, not too intrusive sounds are applied deliberately and rather sparingly in order to not distract 
from the other material. The sound needs to fit in with the context and is arranged in a one-to-one 
correspondence with the related spoken text (or “idea unit” of the text) and pictures so that the information 
can be easily associated with each other. Even spoken text and other sounds can be perceived when presented 
simultaneously, this work suggests to present verbal and related non-verbal auditory information somewhat 
successively due to possible interferences. For the sound to be intelligible and meaningful, the application 
presents them with and shortly after the related text unit and/or picture. That way, text or picture, which are 
certainly more self-contained and self-explanatory, provide the necessary context to enable a clear attribution 
of the sound. 

In order to verify these assumptions and design decisions, one “control chapter” of the application does 
not comply with the design principles as made above. Here, rather ambiguous and intrusive sounds are 
played simultaneously to the spoken text and alongside animated pictures, which is supposed to make 
cognitive load higher, holding the risk of reducing the learners’ capacity for processing relevant information. 

Even though sound can take numerous different forms, such as speech, alarm signals and “beep”-sounds, 
music, and more, considerations in this experiment are limited to the inclusion of environmental sounds with 
the aim to facilitate imagery and a dual coding of information (Paivio, 1986; Thompson & Paivio, 1994). 
That means, environmental sounds included are referred to real events related to the content of the lesson and 
have meaning based on their causal relation to that event. For example, the picture of a cog (replica) at sea 
and its general information is supported by the sound of sailing, that is mainly wind and waves plus a clear 
deep wooden creaking and the flapping of a sail, in order to increase the listeners immersion into the content 
and to make him/her feel the dangers of such journeys. These sounds should mentally refer to and reinforce 
the sound-producing source, which is the cog as wooden - sailing - ship depicted by the picture. Further 
sounds illustrate and strongly highlight the circumstance the cog sank. Hammering on wood (it sank when 
still being under construction), the sound of whistling wind, storm, flood, creaking and breaking wood, and 
crashing depicts the process of the sinking cog during a storm. These sounds are all source-disconnected 
(“off camera”), that is, neither ship-building nor the sinking cog is depicted by the pictures, and should thus 
increase the listener’s imagination. The use of the capstan is illustrated by the sounds of using the capstan, 
quarter turnings, wooden tool, sticks and quarter turnings, panting of men, the latter “off-camera”, which 
intends to make this procedure rather palpable and reinforce the information that this tool was made of wood, 
heavy to use - hard work - and needed to be used by multiple man. 
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2.2 Method 

2.2.1 Measures and Instruments 

The prior knowledge is controlled via a short pre-knowledge questionnaire before instruction. Five open 
questions assess the participants prior knowledge and understanding of basic concepts concerning the 
Hanseatic League and cogs in general. 

Learning performance is gained from learning objectives achieved by the participants, which is measured 
by points given for correctness and/or accuracy of answers. An achievement questionnaire that has to be 
completed directly after instruction tests memory of the material via retention questions (mostly multiple 
choice) and deeper understanding via open and transfer questions. 

Mental effort refers to the cognitive capacity that is allocated to accommodate the demands of the task. 
Due to practical reasons, data is collected through self-reports of the participants using a 7-point likert-scale, 
which is a valid and widely-used tool for measuring mental load as supposed to correlate highly with 
objective measures (e.g. Kalyuga et al. 2000, p. 130). Participants are asked to rate the mental effort they 
needed for the task by translating it into numerical values from 1 (very low) to 7 (very high) directly after 
instruction and after the achievement-test, so that the values indicate mental effort associated with learning 
and with the performance test. 

Task involvement combines measures of mental effort and performance. Presuming that motivation, 
mental effort invested on the task, and performance are positively related, Paas et al. (2005) theorizes that the 
combination of mental effort and performance score is a more accurate measure of motivational effects of 
instructional conditions then subjective rating scales. High performance associated with high effort is called 
high involvement instruction, whereas low-task performance with low effort is called low involvement 
instruction. The computational model of task involvement (respectively learners’ motivation on the task) 
calculates the relative involvement of learners by mapping z-values of mental effort and performance score 
using a particular formula as derived and depicted in Paas et al. (2005, p. 29). 

The “ User Experience Questionnaire” (“UEQ” 1 ), which allows the participants to express feelings, 
impressions, and attitudes towards the application, is utilized to gain further quantitative data that helps to 
interpret (as supposed to mediate) the performance results. In particular, the UEQ is supposed to measure the 
affective and attentional factors the sound might have an influence on. This test uses the scales 
Attractiveness, which assesses general impressions towards the application, and Stimulation, which asses if 
the application is interesting and exciting to use and if the learner feels motivated to further use it. 

2.2.2 Participants and Procedure 

The participants are 38 Digital Media Bachelor students in their 1st semester recruited from the University of 
Applied Sciences in Bremerhaven (Germany). The mean age is 22 years ranging from 18 years to 32. 42% of 
the sample are male students. 

Two different versions of the multimedia-based learning module “A Guided Tour of the Hanseatic Cog” 
are copied onto computers in the computer lab before the experiment. The participants are randomly assigned 
to one of the two experimental groups: 

( 1 ) One group of participants receive a slideshow containing pictures and verbal information in form of 
audio narration (basic-group). 

(2) The same material as in ( 1 ) but additionally supported by corresponding environmental sounds that 
illustrate the content of the lesson (sound-group). 

Participants are not informed about the research question and the differences of the two versions of 
learning material. The test is conducted in three sessions with 11-13 participants each lasting around 75 
minutes. Each student completed the instruction individually using a personal computer and good quality 
headphones, which lasts about 20 minutes. Once the instructional program is finished each participant get the 
first post-test questionnaire, which asks the participants to rate their invested mental effort during instruction, 
followed by the 13 items retention and transfer questions. This questionnaire concludes again asking for the 
invested mental effort on the achievement test. After that, the participants are asked to fill out a second form 
including further two transfer questions and the UEQ. 


1 http://www.ueq-online.org/; (perceived July, 2015) 
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2.3 Results 

2.3.1 Preliminary Study 

Most of the participants have very little prior knowledge about the topic Hanseatic League and cogs as 
indicated by the pre-test questionnaire. However, four participants are excluded from further calculation 
because of high scores in the pre-knowledge test. A homogeneous sample of 34 participants remains for this 
study; having 16 students in the basic-group (group 1), 18 in the sound-group (group 2). Simple t-tests 
indicate that finally both groups do not differ significantly (with p < .05) in basic characteristics, that is, 
pre-knowledge, gender, mean age, and initial motivation regarding the test and the learning topic. 

The reliabilities for the post-test scales are calculated using Cronbach’s Alpha, which reveals satisfactory 
results for the 17 items of the post-achievement test (a=0,78). 

2.3.2 Learning Performance 

As Figure 2 shows, the sound-group’s performance score is higher on the descriptive level in all types of 
question, i.e. open, retention, and transfer questions. However, separate analyses of covariance (ANCOVA) 
with the experimental condition as independent variable, the performance scores as dependent variables and 
the pre-knowledge as covariate (as there is a strong positive correlation between performance and 
pre-knowledge, with r=.56; p=.00), shows that the sound-group did not perform significantly better then the 
basic-group on the retention nor on the transfer part of the post-test. 



Figure 2. Performance score 

In order to verify the design decisions theoretically set in order to prevent cognitive overload, the control 
chapter, which is covered by question 13, is evaluated separately. An analysis of the score of the nine 
retention questions show that the results of question 13a and 13b exceptionally differs from the other results 
in a way that the basic-group considerably outperformed the sound-group (Figure 2). 

Overall, there is no evidence that the inclusion of sound within audiovisual learning material improves 
transfer and retention performance in learning. However, the results suggest that learning performance 
depends on it’s design, such as its intrusiveness and arbitrariness, and it’s integration with the rest of the 
material in a way that sound may impede retention if it is not integrated deliberately. 


2.3.3 Mental Effort 


For the mental effort ratings separate analyses of variance (ANOVA) were conducted using the experimental 
condition as independent variable and the mental effort ratings for the instruction and for the test phase as 
dependent variables. Table 1 shows that the mean of mental effort, which are mapped to a scale from -3 to 
+3, are higher for the sound-group during the test phase, but do not differ significantly. 

Table 1 Mental effort ratings mapped to a scale from -3 to +3 


mental effort ratings learning phase 

N 

M 

SD 

F 

P 

16 

,25 

,68 

1,14 

,43 

18 

0 

1,08 



mental effort ratings test phase 

N 

M 

SD 

F 

P 

1 ) without sound 

16 

,66 

1,01 

4,03 

,12 

2) with sound 

18 

1,11 

,63 
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2.3.4 Learner Involvement 

A regression calculation indicate that mental effort during the test phase may be a positive predictor for the 
performance score (with p=.04), and thus supports the cognitive-motivational viewpoint as depicted by the 
learner involvement construct. Accordingly, the z-values of mental effort ratings and performance score of 
the transfer questions - as being an indicator for germane load investment and “deep” learning - are mapped 
on a Cartesian axis in order to calculate the task involvement of the learner. Figure 3 shows that the sound 
group is clearly in the high involvement area compared to the basic group. 


Performance 



high involvement 

1=0 

• 

mental effort 

-i -ds 

1—0.41 

♦ 

V cls i 


• with sound 

♦ withouk sound 


Figure 3. Learner involvement during test phase 


An ANOVA reveals that the sound-group (M=0,36; SD=0,92) shows a significant higher learner 
involvement during the achievement test than the basic-group (M=-0,41; SD= 1 ,25), with p < .05. 

2.3.5 User Experience 

Even though the sound-group is expected to state higher affective ratings than those of the basic-group, the 
groups did not differ significantly on their mean ratings of attractiveness or stimulation of the learning 
material. 


3. CONCLUSION 

3.1 Summary of Main Findings 

The study does not confirm differences in retention and transfer performance nor in subjective mental effort 
ratings during learning, which suggests that sound, as designed, arranged and coordinated in this work, just 
do not impose additional cognitive load during learning nor have other effects on cognitive processes, which 
benefit or hinder learning. Another possible interpretation is that the advantages concerning motivation (as 
assumed in this study) can be offset by the distraction disadvantages of the environmental sounds, which 
results in no differences in the learning performance between learners that have learned with audio-enhanced 
narratives and the basic version without sound. 

However, a regression calculation confirms that the invested mental effort in the achievement test is a 
predictor for the performance score, that is, the more mental effort the participants invested in answering the 
test questions the better they scored in the test. As a result of the higher performance score and mental effort 
ratings, the study finally shows a significant higher learner involvement of the sound-group, which supports 
the notion of sound having a positive influence on the learner’s motivation to engage in appropriate learning 
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processes. However, as there is no difference in mental effort investment during learning, the results suggest 
that motivational effects may exert its influence only during the subsequent performance test. 

3.2 Discussion 

The study does not identify the mechanisms by which the integration of sound encourage learners to invest 
more effort in answering the test questions. Against expectation, the results reject that the higher involvement 
of the sound-group is attributable to the audio-enhanced narratives being more stimulating and appealing, as 
the UEQ scales reveal. One possible interpretation of the results, even though rather speculative, refers to 
positive imagination effects induced by the environmental sound. The instructional material may have 
encouraged imagery and concreteness of the learning topic and to process the concept of the material actively 
and more deeply, which could have lead to an increase in mental effort investment conducive to learning. 
This notion is supported by participants’ statements that the presentation of sound has made them feel rather 
immersed into the content of the lesson. However, this could be an interesting topic for further investigations, 
which should also include appropriate qualitative measures. It has to be noted that the use of subjective rating 
scales and with it the participants ability to reliably value his/her affective state and invested mental effort 
has clearly its limits. 

It can be concluded that the results do not confirm Moreno and Mayer’s (2000) study that reveal 
detrimental learning effects induced by the incorporation of sound effects into audiovisual learning material. 
The deliberate design and inclusion of relevant sound as it is done in the application used in this experiment, 
does not increase extraneous cognitive load nor has negative effects on learning performance. However, 
practically, the results suggest that sound also risks interfering with the processing of the otherwise essential 
verbal information depending on its design and integration. Therefore, the study concludes in accordance 
with Moreno and Mayer (2000, p. 124): More studies that examine the design of sounds, its integration and 
coordination in multimedia lessons seem to be necessary to get solid results. 

3.3 Limitations 

The study can be seen as a first attempt to investigate the cognitive effects of having sound effects in 
multimedia instruction. It was carried out within the framework of a Master’s Thesis and has thus clearly it’s 
limits - starting from the small sample size of 38 participants. 

It can be assumed that only learners with a basic (at least) medium prior knowledge may profit in general 
from additional environmental sounds. However, the role of prior knowledge (as well as other student 
characteristics, such as spatial ability, age, gender, etc.) has to be subjected to further investigations as it 
couldn’t be provided by this experiment. 

The results and conclusions are also practically very limited as the study only deals with two different 
instructional formats, that is multimedia explanations with or without sound, and with one kind of learner. It 
also entirely neglects to look into different instructional methods, which should be closely connected to the 
choice of media and its format as well as learner characteristics (see e.g. Moreno, 2006). To extend the 
external validity, more research is needed in order to generalize the results to other learning environments, 
also more authentic ones, such as a museum or mobile guides in different contexts, also to different 
instructional methods (e.g. game -based methods), to other learners, as well as to other learning topics. 

With the aim to get clear and solid results, the case study has concentrated on the design and evaluation 
of mainly one aspect of how sound can be utilized in multimedia learning, which is the role of motivation in 
order to improve memory performance. However, sound promises to have many more possibilities to support 
multimedia learning in the same way as sound can take numerous other forms than environmental sounds, 
such as musical sounds or other abstract sounds, which are worth investigating. For example, guiding 
musical themes, leitmotifs, could be a means to orientation and structuring in lessons, as they can be applied 
to repeating events. It can furthermore be assumed that auditory features being distinct from the content of 
the lesson could be also used to elicit and maintain attention and support structuring and selection of the 
content to be further processed in working memory. More on sound’s potential role in instructional software 
can be found in Bishop and Cates (2001). 
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ABSTRACT 

An expansion of the Will, Skill, Tool Model of Technology Integration to include teacher’s pedagogical style is proposed 
by the authors as a means of advancing the predictive power for level of classroom technology integration to beyond 
90%. Suggested advantages to this expansion include more precise identification of areas to be targeted for teacher 
professional development, and the prospect for aligning teaching-with-technology style with student learning styles, in 
order to better serve educational system goals such as student engagement, learning and achievement. Initial findings are 
that pedagogical preference or style with old and new technologies accounts for approximately 30% of level of classroom 
technology integration. The authors contend this is worthy of retaining as a fundamental model improvement. 


KEYWORDS 

Technology integration, predictors, will, skill, pedagogy, access to technology tools. 


1. INTRODUCTION 

Since 2005 the authors have surmised that the Will, Skill, Tool Model of Technology Integration (WST) 
unveiled at the turn of the century (Knezek, Christensen, Hancock, & Shoho, 2000) and shown in Figure 1, 
might be improved by having pedagogical practice (teaching style) “become a frill partner in the model” 
(Hancock, Knezek, & Christensen, 2007, p. 91). A higher order factor analysis (Dunn-Rankin, Knezek, 
Wallace & Zhang, 2004) completed in 2014 indicated that one of the original model’s measurement scales, 
Teaching with Technology (TWT), and a newly developed measurement scale. Emerging TWT, clustered 
together in a higher order domain tentatively labeled Pedagogy (Christensen & Knezek, 2014). This 
prompted the authors to conduct a preliminary test of the integrity of Pedagogy as a separate model construct. 
This paper details the procedures employed and reports on the initial outcomes of testing the Will, Skill, 
Tool, Pedagogy Model of Technology Integration (WSTP). 
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Figure 1. Will, Skill, Tool Model of the impact of technology integration on academic achievement. 


2. DEVELOPMENT OF THE WST MODEL 
2.1 The School Learning Environment 

Learning in school has been recognized as a complex issue for many decades. For example, in 1975, 
Klausmeier and Goodwin identified 37 variables in nine major categories that affect the school learning 
environment. Among these were items such as instructional goals and objectives, student intellectual ability, 
teacher interest and knowledge of subject matter, home-school-community relations, physical space and 
equipment characteristics, and availability of print and audiovisual instructional materials (Klausmeier & 
Goodwin, 1975, p 13.). A complete listing of the variables and categorizations developed by Klausmeier and 
Goodwin are provided in Table 1. 
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Table 1. Variables affecting the school learning environment 

I. Objectives 

1 . Outcomes incorporated 

2. How formulated 

3. How used 

II. Subject matter 

1. Kind; e.g., language arts, music 

2. How organized 

3. How sequenced 

III. Instructional materials 

1 . Kind: printed, audiovisual 

2. Quality 

3. Availability 

IV. Characteristics of the learner 

1. Level of achievement and intellectual abilities 

2. Physical maturity and related psychomotor abilities 

3. Affective characteristics: interests, motives, attitudes, values, emotional 
expressions 

4. Health 

5. Self-concept 

6. Perception of situation 

7. Age 

8. Sex 

V. Characteristics of the teacher 

1 . Knowledge of subject matter, development, learning, and teaching skills 

2. Psychomotor abilities and physical attributes 

3. Affective characteristics; interests, motives, attitudes, values, emotional 
expression 

4. Health 

5. Self-concept 

6. Perception of situation 

7. Age 

8. Sex 

VI. Classroom interactions 

1 . Student-student 

2. Student-teacher 

3. Teacher-teacher 

4. Teacher-Administrator 

VII. Organization for instruction 

1. Elementary level: self-contained classroom, team arrangements, 

departmentalized 

2. Secondary level: departmentalized in separate subjects - self-contained, 
team arrangements; broad fields - self contained, team arrangements 

VIII. Physical characteristics 

1 . Space 

2. Supplies 

3. Equipment, etc. 

IX. Home-school-community relations 

1 . Teachers of particular children - Parents of children 

2. Total school - Total neighborhood 

3. Total school district - Total community 
Note. Adapted from Klausmeir & Goodwin, 1975, p 13. 

2.2 Consolidation of Attributes 

Knezek and Christensen (Knezek, Christensen, Hancock, & Shoho, 2000) created a simplified environmental 
model for student learning in the WST model. In this model the 37 attributes identified by Klausmeir & 
Goodwin (1975) were consolidated into three categories focused on learning in a classroom environment 
aided by the integration of technology. The three-construct WST model is shown in left-hand portion of 
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Figure 1. The authors also developed a second stage of the WST model focused on predicting the impact of 
classroom technology integration on achievement. This was in response to the U.S. Department of 
Education’s 1999 calls for gauging the impact of the infusion of technology into the nation’s classrooms 
(Preparing Tomorrow's Teachers to Use Technology Program, 1999). Also in 1999, the Milken Exchange on 
Education Technology identified seven major categories of variables intended to “... help policymakers 
answer the question, ‘What is the return on the public’s investment in K-12 learning technology?” (Coughlin 
& Lemke, 1999, p.3.8). Since the initial model was introduced, other researchers such as DeMars (2001) 
have identified the need to build effective models to deal with the multiple variables. This second stage of the 
WST model, which focuses on the impact of classroom technology integration on student achievement, is 
shown in the right-hand portion of Figure 1 . 


3. APPLICATIONS OF THE WST MODEL 

Knezek, Christensen, and Fluke (2003 ) found using 1999 - 2001 data that 70% to 84% of teachers level of 
technology integration could be predicted based on measures of Will, Skill, and Tool, while 8-12% of first 
and second grade technology intensive reading achievement was attributable to level of classroom 
technology integration. Morales. Knezek, Christensen, and Avila (2005) examined 10 teacher school district 
samples spanning 2001 - 2005 and found Will, Skill, and Tool accounted for 64 - 83% of the variance in 
technology integration proficiency for practicing teachers. Morales (2006) conducted a transnational study of 
the model using refined measures for seven samples of teachers from Mexico and the USA and concluded 
90% to 96 % of classroom technology integration could be attributed to Will, Skill, and Tool measures for 
these teachers. The total variance explained by the model was more than 90% for all samples, with the trend 
being that Skill was the strongest predictor of technology integration in the USA, whereas Tool was the 
strongest predictor in Mexico (Morales & Knezek, 2007). Petko (2012) used the WST model with self- 
selected measures to determine the impact of constructivist teaching style on the use of digital media in 
classrooms. Five factors accounted for a total of 60% of the variance in the intensity of use of Information 
and Communication Technologies. Among these factors were teacher confidence in their competence, access 
to technology, teacher belief in the power of technology to improve learning and frequency of constructivist 
forms of teaching and learning (Petko, 2012). 


4. UPDATED MODEL SPECIFICATION 

The findings by Petco (2012), as described in the previous section, served as an added impetus for the authors 
of the current work to examine the possible benefits of isolating pedagogical style as a separate construct in 
an extension of the basic WST model. According to the original model shown in Figure 1, the constructs 
Will, Skill, and Tool independently influence the Integration of technology in the classroom. Will is 
conceptually defined as a positive attitude toward the use of technology in instruction; Skill as the self 
perceived confidence and readiness to use technology; Tool is related to the accessibility and use of 
technology; and Integration as a self perceived level of technology adoption for educational purposes. Each 
one of the constructs was operationally defined by several measures: 

Will was defined by two measures: Subscales from the TAC and TAT. 

Skill was defined by four measures: Email, World Wide Web, Integrated Applications, and Teaching with 

Technology (TWT) from the TPSA. 

Tool was defined by Flome access and Classroom access to technology. 

Integration was defined by three measures: Stages of adoption, CBAM, and ACOT, as detailed later. 

The additional construct of Pedagogy is being added to enhance the model. Watkins and Mortimore 
(1999, p. 1 7) offered a definition of pedagogy as ‘any conscious activity by one person designed to enhance 
learning in another’. For the purposes of expanding the WST Model, Pedagogy includes teaching style 
(constructivist, behaviorist, etc.) and in the case of technology integration, the level of confidence teachers 
feel in their use of technologies to enhance learning for their students. 
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5. CURRENT CONSTRUCT INDICATORS 

5.1 Measures of Will 

The Teachers' Attitudes Toward Computers 5.1 Questionnaire (TAC) (Christensen & Knezek, 1998) 
measures seven indices regarding teachers' attitudes. These scales are: FI - Enthusiasm/Enjoyment, F2 - 
Anxiety, F3 - Avoidance/ Acceptance, F4 - Email for Classroom Learning, F5 - Negative Impact on Society, 
F6 - Productivity, and F7 - Semantic perception of computers. The reliabilities for these subscales typically 
range from .87 to .95 with K-12 teacher data. 

The Teachers' Attitudes Toward Information Technology Questionnaire (TAT) (Knezek & Christensen, 
1998) is a semantic differential instrument that measures attitudes toward new information technologies 
including email (variable coded as EMAILT), the World Wide Web (WWWT), multimedia (MMT), 
technology for teacher productivity (PRODT) and technology for classroom learning (PRODCL). 
Reliabilities for these scales typically range from .91 to .98 for K-12 teachers. 

5.2 Measures of Skill 

The Technology Proficiency Self-Assessment (TPS A) (Ropp, 1999) measures technology skills in the 
following areas: E-mail (TPSA_EMT or TPSA_EMAIL), Integrated Applications (TPSA_IA or TPSA_IAT), 
use of the World Wide Web (TPSA_WEB) and Emerging Tools. Reliabilities typically range from .81 to .87. 

5.3 Measures of Tool 

Measures of access to technology Tools include the number of computers available to the teacher in the 
classroom as well as teacher home access to a computer and teacher home access to the Internet. 

5.4 Measures of Pedagogy 

The updated version of the Technology Proficiency Self-Assessment (TPSA), TPSA C21, includes two 
measures of pedagogy. Teaching with Technology (TWT) from the original instrument and. Teaching with 
Emerging Technologies (Emerging TWT) such as smartphones and social media, from the updated TPSA. 
Reliabilities typically range from .87 to .93 (Christensen & Knezek, 2014). 


6. OUTCOME MEASURES 

Three self-reported outcome measures were also included in the battery of instruments administered to 
teachers. One was the Level of Use Questionnaire (CBAM_LOU) based on the Concerns-Based Adoption 
Model/Level of Use scale for diffusion of innovation (Hall, Loucks, Rutherford, & Newlove, 1975), while 
another was the Stage of Adoption of Technology in Education scale (Stages) developed by Christensen 
(1997), based on the earlier work of Russell (1995). The third instrument was based on work of Dwyer, 
Ringstaff, and Sandholtz (1990) for the Apple Classrooms of Tomorrow (ACOT) Project, and is called the 
ACOT Teacher Stages instrument. Each of these is a single-item instrument that cannot be easily checked for 
internal consistency reliability. However, test -retest reliabilities for several groups of K-12 teachers have 
shown reliabilities to typically be higher than .8. Research by Hancock, Knezek and Christensen (2007) 
demonstrated that the three indices used together as a Technology Integration scale produced Cronbach’s 
Alpha = .84. For many research studies using these measures, the authors produce a unidimensional factor 
score based on extracting the “true” score for each teacher across the three measures, and use this as the 
technology integration (outcome) measure. 
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7. RESULTS 

Summary results of the 2011-2014 series of analyses are displayed in Table 2. As shown in Table 2, 
Teaching with Technology (Pedagogy, aka Technological Pedagogical Knowledge) alone can explain 33 
percent of classroom technology integration. Not shown in Table 2 is that the old TWT portion of Pedagogy 
is currently a much stronger indicator of technology integration (beta = .446) than the Emerging TWT (beta = 
.147) based on smartphones and other newer technologies. However, both are significant contributors (p < 
.05). It is anticipated that the Emerging TWT will become more important in the future. 

Other analysis details not shown in Table 2 demonstrated that among the Technology Proficiency Self- 
Assessment (Skill) measures. Email skills contributed very little to the regression prediction (beta = -.017, p 
= .793), once the TWT indicators were included (TWT beta = .446, p < .0005; Emerging TWT beta = .137, p 
= .046), and the contribution of WWW skills was also small (beta = .101; p = .216). Interestingly, Integrated 
Applications (Microsoft Office Tools) turned out to be the strongest predictor (beta = .363, p < .0005) on the 
entire TPSA instrument, but Emerging Tech Skills (beta = .123, p = .049) was also a significant contributor 
to classroom technology integration. 

The Tool measures summarized in Table 2 collectively contributed at least nine percent of level of 
classroom technology integration for this group of educators. Among these. Student Classroom Use (beta = 
.145, p = .015) and Home Hours of Use by the teacher (beta = .159, p = .003) were the strongest contributors. 

Implications of the initial exploratory analyses were that the Will, Skill, Tool Model of Technology 
Integration might benefit from placing a fourth predictive construct for the technology integration model, 
called Pedagogy, into the model. As shown in the first RSQ column of Table 2, the components of Skill, 
Tool, and Pedagogy were able to account for 28 percent, nine percent, and 30 percent respectively, of the 
percentage of the construct Technology Integration for the 2014 data set in this study. In addition, as shown 
in the last column of Table 2, a parallel school district data set from 2011 that also included measures of Will 
(Teachers Attitudes Toward Computers, TAC) (Christensen & Knezek, 2009) in the professional 
development assessment battery of instruments administered to 1648 teachers, produced comparable 
regression coefficients to the 2014 study for the three components identified in the data set. Further analysis 
of the 2011 data set indicated that 29 percent of Technology Integration could be attributed to the scales 
comprising Will. The expanded WST Model (WSTP) is shown in Figure 2. 

Over interpretation regarding the strength of each of these predictors is cautioned because some portion of 
the variance predicted by one scale set (for example. Skill) would likely also be able to be predicted by 
another scale set (for example. Pedagogy), and in fact the sum of the RSQs for the four model components in 
the second column of Table 2 is more than 100 percent. Nevertheless, these exploratory findings provide 
hope that a formal structural equation modeling (SEM) approach to testing the WSTP model might confirm 
the predictive power of a four-component model of technology integration to be beyond the previously 
reported 90 percent (Morales, 2006; Morales & Knezek, 2007; Knezek & Christensen. 2008), and 
approaching 100 percent. The authors plan to test this conjecture when the volume of teacher data acquired 
covering all four components of the model, and using the new TPSA, surpasses n = 1500. 


Table 2. Prediction ability of Will, Skill, Tool, and Pedagogy indicators 


Model Component 

Indicator 

RSQ Current 
data set (2014) 
(n = 466) 

RSQ - Prior data 
set (2011) 

(n= 1648) 

Will 

TAC Subscales 2 (Tech Comfort), 6 
(Utility), 8 (Absorption) were sig. 


.29 

Skill 

TPSA 1 (Email), 2 (WWW), 3 
(IntApp) 

(Email was NS) 

.28 

.29 

Tool 

Home use, student use, access 

.09 

.15 

Pedagogy 

TPSA Teaching with Tech 

.33 

.30 
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Figure 2. Expanded will, skill, tool model of technology integration, including pedagogy. 


8. DISCUSSION 

Previous studies (eg. Morales, 2006) found that the strength of the regression-style coefficients for the major 
model components varied depending upon the local context in which the teachers providing the data resided. 
For example, for the Morales (2006) study, level of technology-using Skill development was found to be the 
strongest predicator of technology integration for teachers in the U.S., while for teachers in Mexico, extent of 
access to technology Tools was the most important predictor. The implication is that the domain in which the 
local environment has the greatest variability becomes the strongest predictor. The authors of the current 
paper have found from other studies (Christensen & Knezek, 1999) that for teachers who are near the highest 
level of technology integration, it is the Will to push forward that becomes the greatest predictor. By 
following these analogies, we can infer that among the 2014 and 2011 data sets featured in this paper, 
pedagogical style (Pedagogy) is likely to have the greatest variability among the teachers surveyed, since it 
accounted for 33% (2014 data) and 30% (201 1 data) respectively, of teacher’s level of technology integration 
(see Table 2). The leadership of school districts might wish to provide alternative professional development 
activities for teachers from different pedagogical schools of thought, in order to better support this diversity. 


9. CONCLUSION 

Initial testing of the expanded Will, Skill, Tool Model of Technology Integration indicates that Pedagogy as a 
predictive construct has empirical support from recently-gathered data and promises to bring new 
perspectives to the realm of initiatives for technology-intensive teacher professional development. In addition 
to enhancing the ability of the WST model to predict beyond 90% of a teacher’s level of technology 
integration in his or her classroom, it is likely that elucidating the importance of pedagogical style of the 
teacher will bring more focus on aligning teaching with the preferred learning styles of students. This could 
enable educational practitioners to push beyond the 8-12% per year of positive impact on student 
achievement previously identified as attributable to level of classroom technology integration. 
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ABSTRACT 

This paper briefly discusses four measurement challenges of data science or ‘big data’ in educational assessments that are 
enabled by technology: 1. Dealing with change over time via time-based data. 2. How a digital performance space’s 
relationships interact with learner actions, communications and products. 3. How layers of interpretation are formed from 
translations of atomistic data into meaningful larger units suitable for making inferences about what someone knows and 
can do. 4. How to represent the dynamics of interactions between and among learners who are being assessed by their 
interactions with each other as well as with digital resources and agents in digital performance spaces. Because of the 
movement from paper-based tests to online learning, and in order to make progress on these challenges, the authors 
advocate the restructuring of training of the next generation of researchers and psychometricians to specialize in data 
science in technology enabled assessments. This call to action stemmed from discussions at EDUsummlT 2013, which 
will be published in depth in a special issue of Education and Information Technologies. 
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1. INTRODUCTION 

Assessment and learning analytics challenges have dramatically increased since new digital performance 
affordances, user interfaces, and the targets of technology-enabled assessments have become more complex. 
The increased complexity is due in part to technology’s capabilities and roles in presenting interactive 
learning experiences and collecting rich data (de Freitas, 2014; Quellmalz et al., 2012) which is leading to the 
infusion of data science methods and techniques into learning and behavioural science research (Gibson & 
Knezek, 2011; Kozleski, Gibson, & Hynds, 2012). These changes require new quantitative methods as well 
as a reconceptualization of mixed methods (Tashakkori & Teddlie, 2003) that include domain experts as well 
as stakeholders in the construction of knowledge of such complex systems. 

In technology-enhanced assessments, the emergence of ‘big data’ - which are defined as data with a large 
numbers of records, of widely differing data types, that are rapidly collected for immediate action (IBM, 
2015; Margetts & Sutcliffe, 2013) - underscores the need to develop assessment literacy (Stiggins, 1995) in 
teachers, learners and other audiences of assessment. Assessment literacy has become more important than 
ever for understanding how technology influences and impacts assessment types and processes and especially 
for developing confidence in creating and analysing arguments from evidence, based on a user’s current 
understanding of validation (Black, Harrison, Hodgen, Marshall, & Serret, 2010). 

The paper discusses the main challenges associated with applying data science methods in educational 
assessment to address a digital media assessment’s psychometric properties; time sensitivity; digital 
performance and the problem space for analysis; the hierarchy of tasks, turns and translations between 
different levels and the dynamics of interrelationships in assessment systems. The OECD PISA plan for 
assessing CPS is used as an example to explain these challenges in relation to a complex problem space. The 
paper then illustrates with a learning analytics case that shows how the identified challenges have been 
addressed in the development of assessments. 
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2. BACKGROUND 

There is uncertainty as to whether and how four different perspectives on assessment - providing feedback 
information, supporting improvement decisions, identifying the degree of engagement and understanding, 
and making value judgments - can co-exist to the benefit of learners (Webb, Gibson, & Forkosh-Baruch, 
2013). Even with the increased possibilities that IT provides there is not yet a way to say confidently that the 
multiple purposes for which some assessments have been used (Mansell, James, & the Assessment Reform 
Group, 2009) can or should be supported through the same assessment systems. This is because the impacts 
of some purposes interact with the validation processes of others (Messick, 1994). Therefore in considering 
assessment design for multiple purposes for example for formative as well as summative purposes, users 
need to examine those impacts carefully in order to minimise negative consequences on learning and 
learners. 

Developing theory for the application of data science methods in educational research is important for 
two primary reasons. First, assessment of virtual performance presents new challenges for psychometrics 
(Clarke-Midura & Dede, 2010; Ifenthaler, Eseryel, & Ge, 2012; Quellmalz et al., 2012). Secondly, new tools 
are needed for discovery of patterns and drivers in complex systems for working with ‘big data’ in 
educational research preparation and practice (Gibson, 2012; Patton, 2011). Indicators of progress in theory 
development would be an increase in research exploring and articulating the use of data science methods in 
learning analytics to improve learning and achievement; and the expansion of methods beyond traditional 
statistics and qualitative approaches in educational research, to include data mining, machine learning, and in 
general, the methods of data science. 


3. PSYCHOMETRIC CHALLENGES IN THE ERA OF BIG DATA 

Psychometrics is the branch of psychology that deals with the design, administration, and interpretation of 
quantitative tests for the measurement of psychological variables such as intelligence, aptitude, and 
personality traits (“Psychometrics,” 2014). A good psychometric test is “internally consistent, reliable over 
time, discriminating and of demonstrated validity in respect of its correlations with other tests, its predictive 
power and the performance of various criterion groups. It also has good norms” (Kline, 1998, p.92). 

Until recently, the field dealt almost exclusively with the construction and validation of measurement 
instruments such as questionnaires, tests, and personality assessments. However there is now a need to 
expand to include highly interactive digital learning and adaptive test experiences, such as the OECD PISA 
assessment of CPS. In brief, PISA is a triennial international survey that aims to evaluate education systems 
worldwide by testing the skills and knowledge of 15 -year-old students in order to determine the extent to 
which they can apply their knowledge to real-life situations and hence are prepared for full participation in 
society. To constrain the quite complex variables that would be involved if the collaboration was among a set 
of real people, the OECD assessment utilizes the computer to play roles as collaborators in a virtual 
performance assessment (Clarke-Midura, Code, Dede, Mayrath, & Zap, 2012). The PISA assessment plan 
incorporates a complex behaviour space that illustrates some of the new demands on psychometrics. 

The challenge with technology enabled assessments is to evolve the procedural foundations of 
psychometrics, which until recently have been primarily based on population statistics and static snapshots of 
data. The new foundation outlined here highlights the need to include time sensitivity, digital performance 
space relationships, multiple layers of aggregations at different scales, and representations of the dynamics of 
a complex behaviour space (Gibson & Jakl, 2013; Quellmalz et al., 2012). 

3.1 Time Sensitivity 

In the OECD assessment, time is controlled as a boundary variable of the test and the computer is used to 
prompt the test taker to ‘move on’ when the evidence rule system detects that the student needs to be rescued 
from an unproductive problem-solving path. The decision to redirect appears natural to the situation because 
the computer is playing the role of one or more collaborators, so the suggestion to move on comes from a 
simulated peer. This situation illustrates that a technology-enabled assessment might well give the student 
perceived or actual control over time, compared to an assessment that only displays test item prompts in a 
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timed test. In some virtual performance assessments, time is open-ended, and the use of item resources (e.g. 
in what order, with or without returning to the resources multiple times, time spent with each resource, timing 
of the appropriate use of a resource, and total time to utilize the appropriate resources to accomplish the task) 
all may be critical to the classification of the learner’s response (Gibson & Jakl, 2013; Stevens & 
Palacio-Cayetano, 2003). 

The OECD assessment solves the time sensitivity problem by parsing time into critical events and then 
monitoring the event patterns to detect the level of evidence of the competencies in the domain model (see 
Table 1). This is a form of time segmentation, because some events cannot happen until other events have 
occurred (e.g. establishing and maintaining team organisation must occur after establishing a shared vision, 
and while maintaining that vision and taking appropriate action to solve the problem). A planned sequence of 
activities and timed release of testing resources, known in game -based learning as a ‘branching storyline’ 
(Aldrich, 2005) is a method for controlling the evolution of a process. 

Table 1. Domain model for assessing collaborative problem solving 



(1) Establishing and 
maintaining shared 
understanding 

(2) Taking appropriate 
action to solve the problem 

(3) Establishing and 
maintaining team 
organisation 

(A) Exploring and 
Understanding 

(Al) Discovering 
perspectives and abilities of 
team members 

(A2) Discovering the type of 
collaborative interaction to 
solve the problem, along with 
goals 

(A3) Understanding roles to 
solve problem 

(B) Representing and 
Formulating 

(B1) Building a shared 
representation and 
negotiating the meaning of 
the problem (common 
ground) 

(B2) Identifying and 
describing tasks to be 
completed 

(B3) Describe roles and team 
organisation (communication 
protocol/rules of engagement) 

(C) Planning and 
Executing 

(Cl) Communicating with 
team members about the 
actions to be/ being 
performed 

(C2) Enacting plans 

(C3) Following rules of 
engagement, (e.g., prompting 
other team members to 
perform their tasks.) 

(D) Monitoring and 
Reflecting 

(D1) Monitoring and 
repairing the shared 
understanding 

(D2) Monitoring results of 
actions and evaluating 
success in solving the 
problem 

(D3) Monitoring, providing 
feedback and adapting the 
team organisation and roles 


Other problem-solving contexts, such as coordination of group actions needed for scientific inquiry and 
experimentation, require simultaneous actions mixed with sequences of actions. The classification system of 
the assessment has to handle patterns of simultaneous and sequential interactions in order to make valid 
links to time-sensitive evidence rules within the conceptual assessment framework (CAF), which is a key 
component of evidence centred design (Mislevy, Steinberg, & Almond, 1999), an approach that is becoming 
increasingly prominent in assessment design and on which this analysis is based. The CAF has three core 
components: the student model, task model and evidence model (Mislevy et ah, 1999; Mislevy, Steinberg, & 
Almond, 2003) within and among which the time sensitive relationships adhere. 


3.2 Digital Performance Space Relationships 

A learning experience entails a designed structure of knowledge and action (Jonassen, 1997) and when that 
experience is interactive and digital there are many measurement challenges (Quellmalz et al., 2012). The 
emerging varieties of network analysis (e.g. social networks, visualization, artificial neural networks, 
decision trees) have arisen as new analytical tools and methods for understanding the structural relationships 
in technology-enhanced learning (Choi, Rupp, Gushta, & Sweet, 2010; Shaffer et al., 2009). In addition, the 
traces of knowledge and action (i.e. the actions, communications and products) created by a learner during 
the course of interacting with a digital learning application bear a relationship to that person’s mental 
representations of the problem (Newell & Simon, 1972) and the knowledge and capability they acquired. 
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accessed and utilized during the interaction (Pirnay-Dummer, Ifenthaler, & Spector, 2010; Thagard. 2010). 
This set of ideas are referred to here as ‘ digital performance space relationships' which will be shown to be 
similar to ‘items’ and ‘constructs’ in classical test theory. 

An interactive digital performance space can support several scenarios, each with one or more 
classification challenges for inferring what the test taker knows and can do. In the OECD assessment, for 
example, the scenarios presented to the student are designed to sample the digital performance space 
construct of ‘collaborative problem solving.’ Each scenario allows the classification of the test taker into one 
or more cells of a matrix created by the intersection of three stages of ‘collaboration’ with four stages of 
‘problem-solving’ (Table 1). In classical test theory, the ‘construct’ plays a similar role to the digital 
performance space; several test items are used to make multiple measures of the construct. A review of the 
historical idea of a valid construct is helpful for making the bridge from classical testing to the digital age. 

A valid construct was thought of as an inductive summary and as part of a series of validity investigations 
that included concurrent, predictive and content considerations. In addition, the construct can change and 
become more elaborated over time, as Cronbach noted (Cronbach & Meehl, 1955): 

When a construct is fairly new, there may be few specifiable associations by which to pin down the 
concept. As research proceeds, the construct sends out roots in many directions, which attach it to more and 
more facts or other constructs. 

Finally, the construct acquired validity through the idea of a nomological network which is a collection of 
overlapping mappings from (a) observable properties or quantities to one another; (b) different theoretical 
ideas to one another, or (c) theoretical constructs to observables (ibid). A single mapping might include 
examples of all these relations, as a construct might be a complex set of factors that interact with one another. 
The idea of a network of ideas and relationships was a fairly abstract philosophical idea in the 1950’s but 
today has a concrete meaning that has become known as network theory in social science (Borgatti & Halgin, 
2011) and network analysis in computational sciences, both of which are applied graph theory from 
mathematics (Brandes & Erlebach, 2005). This history outlines a bridge of ideas that carries forward into 
today when digital media learning spaces can record a network of traces of the actions of a learner. 

Digital media learning presents problems as well as prompts for learner performance (e.g. 
problem-solving, collaboration) in a space that is characterized by hyperlinked resources that can be 
represented as nodes and relations in a network (Clarke-Midura et al., 2012; Quellmalz et al., 2012; Stevens, 
2006). As a learner uses such a space to learn and perform (e.g. interacting with the resources to solve a 
problem, adding new information, re-arranging resources into new relationships) a new network can be 
created that represents the learner’s response, a time-specific performance path through the digital 
performance space (Ifenthaler et al., 2012). The learner’s performance network is a constructed knowledge 
structure that needs to be taken into account in assessment (Gijbels, Dochy, Van den Bossche, & Segers, 
2005). The digital performance space and the constructed knowledge structure of the learner hold the same 
kind of relationship as the nomological network does to a demonstrated construct; the digital performance 
space holds the learning designer’s view of the construct (e.g. what it means to act like a scientist in a given 
situation) and the constructed knowledge structure (e.g. what the learner did in this instance) holds evidence 
of the processes and products of knowing and doing. 

The terms of the nomological network inference, which underpins a claim of construct validity, bear a 
similarity to the rules of a chain of a reasoned argument, which can lead to a claim concerning what a learner 
knows and can do as used in Evidence-Centered Design (ECD). In ECD, an argument has constituent claims, 
data, warrants and backing and must take account of alternative explanations. In a nomological network by 
comparison, there are observations, ideas and relationships and a chain of inference must be used in order to 
establish a claim that a particular test is a measure of the construct. 

The relationships and nodes of a network representation of the traces of learner interactions can be 
compared to the digital performance space resources and relationships to enable inferences about what the 
learner knows and can do (Al-diban & Ifenthaler, 2011; Quellmalz, Timms, & Schneider, 2009). Network 
measures such as similarity, centrality, clusters and pattern matching are used in such inferences, where the 
patterns of the network imply functional and structural connectivity (Sporns, 2011). Digital performance 
space relationships examined with time sensitive network analysis has increased the ability of research to 
characterise and make comment on processes, products, knowledge and know-how, and their complex 
entanglements in authentic performance settings. 


95 



ISBN: 978-989-8533-43-2 ©2015 


3.3 Layers of Aggregations and Translations 

In the OECD assessment of CPS, aggregations of events into tasks takes place in a hierarchy that begins at 
the top with a scenario and ends within each task of the scenario at the level of a ‘turn’ - a game-based 
learning concept that updates the state of the scenario based on the learner’s input. 

Each problem scenario (unit) contains multiple tasks. A task, e.g., consensus building, is a particular 
phase within the scenario, with a beginning and an end. A task consists of a number of turns (exchanges, 
chats, actions, etc.) between the participants in the team. A finite number of options leading onto different 
paths are available to the participants after each turn, some of which constitute a step towards solving the 
problem. The end of a task forms an appropriate point to start the next task. Whenever the participants fail to 
reach this point a ‘rescue’ is programmed to ensure that the next task can be started (PISA, 2013). 

With this hierarchy in mind (e.g. scenarios containing tasks that contain turns) the challenge of 
aggregating with time sensitivity and translating from one level of analysis to another can be addressed with 
moving averages, sliding time windows, and event recognition. The OECD uses event recognition, in which 
an action, communication or product of the test taker triggers a reaction by the test engine to update the 
scenario, which might include rescuing the test taker. In a moving average, some window of time is selected 
(e.g. every second, or after every three turns) and an average is performed to form an abstracted data layer 
that preserves some of the shape of the data movement over time. In the sliding time window (Choi et ah, 
2010; Han. Cheng, Xin, & Yan, 2007), a combination of event recognitions and moving averages, or some 
configuration of either, might be performed and then used as an abstracted data layer. In the example case 
summarized below, for example, the time stamps of every action were subtracted from each other to compute 
duration, which was then applied to each action, to nearby action-pairs and to action-ngrams (motifs) for 
further analysis. 

Within any slice of time, or when comparing two or a few slices of time, standard statistical procedures 
and aggregations apply (e.g. means testing, correlations, regressions), but when high resolution data is 
involved (e.g. many data points per record per unit of time) and where there are complex aggregations (e.g. 
widely varying sources of data and different units of measure) then data mining techniques are more 
applicable. Of note, regression techniques in data mining are not equivalent to the same methods in statistics, 
even though the terms sound and look the same. In data mining, regression represents a search within a 
complex nonlinear space for patterns and representations of structure and causal dynamic relationships, rather 
than the reduction of error of a linear model (Schmidt & Lipson, 2009). Thus, aggregations in the two 
approaches are also of different lineage and need to be considered as separate entities with separate 
representational functions, meaning and purposes (Bates & Watts, 1988). 

3.4 Representations of Dynamics 

Systems dynamics (Bar-Yam, 1997; Sterman, 1994) involves a mathematical modeling technique for 
framing, understanding, and discussing the issues of time, digital performance space relationships and 
aggregation-translation in highly interactive technology-enhanced assessments. Field experiments with 
systems dynamics methods have for example, focused on mid-level model-based theory building in an 
assessment context (Kopainsky. Pirnay-dummer, & Alessi, 2010). The process of building a model from 
snapshots of a dynamic system is called a ‘nonlinear state space reconstruction’ (Sugihara et al., 2012). In 
such a state space equivalent to a network all the data falls within a finite band or manifold of behaviour. 
That is, every state of the system will be in one of the spaces created by the finite possibilities for each 
variable at some point in time. Such reconstructions of the underlying manifold governing the dynamics of a 
system can map to and uncover the causal relationships in a complex system (Schmidt & Lipson, 2009) 
including those that support inferences concerning what a user knows and can do. 

Visualizing the current status of a learner’s progress on an assessment is an example of representing a 
state of a dynamic system, as is visualizing the progress of the learner in relation to a domain model driving 
the assessment’s evidence collection processes. The Khan Academy (Khan, 2011) for example, charts 
progress in learning mathematics or science content against a visualization of the content hierarchy. If the 
learner has mastered division, a visual tree shows how mastery fits with addition and subtraction and allows 
access to the next higher level of math skill. More dynamic and fine-grained visualizations are also possible, 
for example, that would trace the process steps of a solution, or document the details of a constructive 
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process. Visualizations can aide pattern discovery involving both nonverbal and verbal expressions; for 
example, from bodies of text, from online student discussion forums, and from cognitive and mental model 
representations (Pirnay-Dummer et al., 2010). 

To date the developments in learning analytics that provide visualisations of learning traces for learners 
and teachers have been represented by learning analytics dashboards. Such dashboards have been developed 
that keep track of time, social interactions for insights into collaboration, the use of documents and tools, and 
the artefacts produced by students (Verbert, Duval, Klerkx, Govaerts, & Santos, 2013). While these 
dashboards currently fall far short of the detailed traces of assessment data that are possible to create, even 
these more limited opportunities for analysing their learning have been found to support learners' reflection 
and improve self-assessment as well as increasing course satisfaction (Verbert, et al., 2013). 

Examples of the more highly detailed traces are readily found in serious games, as well as casual games 
that are designed to be immersive and emotionally engaging rather than a simple pastime (Aldrich, 2005). In 
these game -based examples, the high -resolution feedback is always on, giving the player an up-to-date view 
of progress, hints about upcoming challenges, and a view to the longer-term goal (Prensky, 2001). Clearly 
educators and researchers might want to promote to policymakers the importance of researching the methods 
and impacts of presenting visualisations of data to teachers and learners along with developments in data 
processing that will better enable judgements of student performances. 

Perhaps the biggest unresolved issue of representation of collaborative learning (and perhaps any learning 
progress during a complex process) is how to represent the moving and evolving quality of change over time. 
‘Movies’ of dynamic educational processes have not yet been documented in many cases, and if existing, 
have not been widely disseminated into common practice. This lack of a practice base and experience 
hampers theory as well as practice in technology-enhanced assessments, and points to the need illustrated by 
the case in the next section, for future research and practice to create a shared understanding of the methods 
of data science in educational research. 


4. CASE STORY: VIRTUAL PERFORMANCE ASSESSMENT 

A case story illustrates how technology enabled educational assessment can produce a large number of 
records, how time and process can be an included mediating factor in analysis and how machine learning and 
data mining methods are needed to support the rapid simultaneous testing of multiple hypotheses. 

A game-based assessment of scientific thinking was created at Harvard (Clarke-Midura et al., 2012) and 
analysed by one of the authors (Gibson & Clarke-Midura, 2013) to ascertain the abilities of middle school 
students to design a scientific investigation and construct a causal explanation. A summary of the data 
science findings and issues included the observation of two of the three aspects of big data: volume 
(~ 821,000 records for 4000 subjects, or 205 records per subject); and variety of data (user actions, decisions 
and artifacts provided evidence of learning and thought processes). The third element of big data, velocity, 
was less important in this case; because the flow of data was not used in near -real time to give hints, correct 
mistakes, or inform the learner during the experience, so the data was streamed off to storage for later 
analysis. 

This case illustrates several of the features of big data in educational assessment. First, the context was 
captured along with the learner action, decision, and product, but that context needed to be effectively 
constructed from the smallest items of data into larger clusters of information. For example, a data element 
named ‘opened door’ by itself was relatively meaningless compared to knowing that it was a particular door, 
opened after another significant event such as talking to a scientist. Thus, patterns of action were transformed 
into n-grams (Scheffel, Niemann, & Leony, 2012) or motifs , which then became the transformed units of 
analysis. This concept of the unit of analysis containing the semantic, time and space contexts for lower 
levels of aggregation may be a new methodological requirement of digital assessments, and needs further 
study. 

Second, as a large number of users traverse through the network of possibilities in a digital performance 
space, key movements of the population within the network can be counted and then used as the basis for 
empirical prior probabilities which assist in creating Bayesian inferences about the scientific 
problem-solving path-maps of learners (Stevens, Johnson, & Soller, 2005). In particular, each pathway in 
such a network can be further characterized or specified with a predictive nonlinear mathematical 


97 



ISBN: 978-989-8533-43-2 ©2015 


relationship (Gibson & Clarke-Midura, 2013), for example, found through symbolic regression an 
evolutionary machine learning technique (Schmidt & Lipson, 2009). Or, alternatively an association rule 
network can be created that distinguishes user action patterns and motifs according to the prevalence of 
utilizing one resource compared to another. For example, if 100% of the population goes to resource 3 after 
resource 1 (skipping over and not utilising resource 2), then with a very high probability, if the sample is a 
good sample of the greater population, the next user entering the system will follow that path and the 
inference system can make a highly probable educated guess about what the person now using resource 1 
will do next. 

The third feature is that the complex set of relationships in various analyses such as those just mentioned, 
bear a structural relationship to something meaningful about the digital performance space as outlined above. 
For example, a cluster analyses can reveal that some resources are critical to success and others are ignored 
and not important to the most successful learners (Quellmalz et al., 2012) or a network visualization can 
highlight how people relate to each other or to a task such as quoting and using scientific resources (Bollen et 
al., 2009). 


5. CONCLUSION AND IMPLICATIONS FOR TEACHING AND 
LEARNING 

This paper has introduced four challenges of big data in educational assessments that are enabled by 
technology: how to deal with change over time and time-based data; how a digital performance space’s 
relationships interact with learner actions, communications and products; how layers of interpretation are 
formed from translations of atomistic data into meaningful larger units; and how to represent the dynamics of 
interactions between and among learners who are being assessed by their interactions in digital performance 
spaces. The article linked the big data challenges to solutions offered in the OECD PISA assessment of 
collaborative problem solving, and then reviewed some of the same issues by briefly summarizing a 
particular case. 

The challenges and issues discussed in this article reveal the requirements for developments in theory as 
well as some of the practical challenges that will need to be overcome if educators are to achieve the vision 
of providing learners and teachers with a ‘quiet assessment’ system in which the impact can be turned up at 
the request of learners and teachers as they seek to understand the progress of learning. This joint approach 
which emphasises assessment AS, FOR and OF learning (Bennett, 2010) is discussed further in (Gibson & 
Webb, 2015) and (Webb & Gibson, 2015) 

In moving forward to embrace the opportunities that could be provided by technology enhanced 
assessments the challenges that remain to be addressed must not be underestimated before educators can use 
automated assessments of complex skills and understanding with confidence. 
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ABSTRACT 

This research study aims at evaluating the suitability of SimReal+ for effective use in teacher education. SimReal+ was 
originally developed to teach mathematics in universities, but it is has been recently improved to include school 
mathematics. The basic idea of SimReal+ is that the visualization of mathematical concepts is a powerful technique to 
enhance the understanding of mathematics. The study uses a set of quality criteria, and a survey questionnaire with 
open-ended questions to evaluate the suitability of SimReal+ in teacher education. The statements of the survey are 
categorized in five main groups of criteria that include technical usability, pedagogical usability, mathematical content, 
assessment issues, and teacher education considerations. Additional issues are programming mathematical visualizations, 
simulations using templates, and flipped classroom. Conclusions and some recommendations are drawn from the results 
to improve the design of SimReal+ to better suit the needs of students and teachers in mathematics education. 


KEYWORDS 
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1. INTRODUCTION 

The Centre for Research, Innovation and Coordination of Mathematics Teaching (MatRIC) has been 
established in 2014 to create and disseminate knowledge and experience that will guarantee high quality 
learning opportunities in mathematics for Norwegian students. This work is situated within the simulation 
and visualization workgroup of MatRIC, which seeks to support a network of mathematics teachers 
collaborating in the production of digital simulations and visualizations by means of SimReal+ to support 
learning processes. SimReal+ is a digital tool that was developed to teach a wide range of mathematical 
topics in higher education. Recently, SimReal+ has been extended to include school mathematics, but the 
suitability of the tool has not yet been evaluated for use in teacher education. According to the New National 
Curriculum in Norway, being able to use digital tools in mathematics education involves using the tools for 
visualization, problem solving, simulation and modeling. However, mastering technical issues of digital tools 
does not automatically make SimReal+ pedagogical usable for teacher students in terms of motivation, 
variation, student autonomy, individualization. Moreover, the content provided by SimReal+ must be 
mathematically sound and correct, and help students gain knowledge that is otherwise difficult to acquire in a 
school environment. Furthermore, assessment issues in terms of feedback must be considered to support the 
learning process. Hence, adaptability of digital tools to a teacher education context is a complex issue that 
needs to be evaluated on the basis of criteria that are pertinent to pedagogical software. Clearly, success in 
mathematics-related subjects does not automatically guarantee success in teacher education. The paper uses 
an instrument to assess the suitability of SimReal-t- for teaching and learning school mathematics. To achieve 
this goal, SimReal+ was used in a course on digital tools in mathematics education. The work was carried out 
in two steps. Firstly, teaching and exercise activities over a period of two weeks were performed. The 
activities included mathematical issues related to trigonometry, programming issues, and diverse teaching 
material made online. Secondly, the students were asked to evaluate SimReal+ using a set of criteria, and 
mixed data collection and analysis methods. The criteria were categorized in five groups that include 
technical usability, pedagogical usability, mathematical content, assessment issues, teacher education 
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considerations, and other issues such as programming and flipped classroom. Open-ended questions allowed 
the students to express in their own words what they think about different issues of SimReal+. Finally, the 
article suggests some recommendations that help placing SimReal+ as an integral part in teacher education. 


2. SIMREAL+ 

SimReal+ is an interactive learning tool for teaching and learning mathematics for a range number of subjects 
related to mathematics such as engineering and physics. The basic idea of SimReal+ is that visualizations are 
powerful mechanisms for learning mathematics and explaining difficult topics. According to Arcavi (2003), 
visualization is the ability to use and reflect upon pictures, graphs, animations, images, and diagrams on 
paper or with digital tools with the purpose of communicating information, thinking about and advancing 
understandings. There is a huge interest in visualization in mathematics education (McKenzie, & Clements, 
2014; Presmeg, 2014). Textbooks are filled with pictures, diagrams, and graphs. Graphing calculators have 
become integral part of mathematics education in secondary schools. Digital tools based on visualizations, 
such as GeoGebra and Geometer’s Sketchpad, are in use in secondary and university mathematics 
classrooms. However, there is little empirical support for the use of visualizations in educational settings 
(Macnab, Phillips, & Norris, 2012). SimReal+ as a visualization tool uses a graphic calculator, video lectures, 
video streaming, video and interactive simulations to teach mathematics. It also provides exercises and 
applications in various areas of mathematics and physics at different educational levels (Brekke, & Hogstad, 
201 1 ). Figure 1 shows the main components of SimReal+. 



There are three research studies on SimReal+ and its effect on teaching and learning mathematics. Two 
studies focused on teaching mathematics in higher education (Brekke, & Hogstad, 2010; Hogstad, 2012). 
These studies report on positive attitudes towards the use of SimReal+ and its suitability and usefulness in 
difficult and abstract mathematical areas. Students considered SimReal+ as a positive addition and 
supplement to ordinary teaching. Students encountered few challenges. The third study addresses the use of 
SimReal+ in teacher education (Cury, 2012). It reports on positive students’ attitudes towards the use of the 
tool in an upper secondary school, but some students did not found visualizations very useful, and that 
integrating the tool into the educational curriculum was not simple. These studies used quantitative and 
qualitative methods to assess students’ perceptions of SimReal+ such as interviews and surveys. 


3. THEORETICAL FRAMEWORK 

A wide range of theoretical approaches can be used to address the integration of digital tools in education 
(Drijvers et al. 2010). No one of these approaches is ready-made for the purpose of this work, but according 
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to Cobb (2007), elements of different theoretical perspectives can be adapted for the concerns of a research 
study as a source of ideas. Hence, the framework used for evaluating SimReal+ is rooted in four approaches 
that are particularly pertinent to this study. The first one is the instrumental approach and the distinction 
between artefact and instrument, and how to transform an artifact to an instrument through the processes of 
instrumental genesis (Rabardel, 1995; Trouche, 2004). Second, the anthropological approach and the 
Task-Technique-Theory triad to capture the relationships between tool techniques and conceptual 
understanding, on the one hand, and tool techniques and paper-and-pencil techniques, on the other hand 
(Chevallard. 1995). Third, the Theory of Didactical Situations, and the tool’s potential for providing 
feedback, which is an essential condition for supporting learning (Brousseau, 1997). The fourth theoretical 
background is based on usability issues and related research work on evaluating digital tools in mathematics 
education (Bokhove, & Drijvers, 2010; Hadjerrouit, & Bronner, 2014). As a result, five groups of evaluation 
criteria are derived from this theoretical background: Technical usability, pedagogical usability, assessment 
issues, mathematical content, and teacher education considerations (Hadjerrouit, & Bronner, 2014). 
Technical usability as defined by Nielsen (1993) is a self-evident requirement for any digital tool in 
mathematics education. In many cases, however, the impact of technical usability on learning is limited when 
it comes to pedagogical use of the tool. In addition, the functionality of a particular tool does not always 
result in pedagogical opportunities, which will only be visible when an explicit pedagogy guides the design 
and use of the tool in classroom (Burden & Atkinson, 2008). Nokelainen (2006) expanded the concept of 
technical usability to include pedagogical usability criteria, such as learner control, collaboration, variation, 
motivation, differentiation, flexibility, and feedback. Hadjerrouit (2010) stated that both aspects of usability 
are closely related to each other. In addition, the assessment dimension has emerged as one of the most 
important criteria for evaluating the quality of digital tools. Assessment is associated with the tool’s feedback 
and builds the ground for formative assessment. Another criterion is the inclusion of mathematical content in 
terms of correctness of mathematical concepts, notations, and symbols (Artigue, et al, 2009; Bokhove, & 
Drijvers, 2010). Equally important is the congruence between the tool’s features and paper-and-pencil 
techniques to facilitate mathematical reasoning. Finally, it is important to evaluate whether the tool is 
appropriate in teacher education, takes into account the requirement of adapted education, and enables the 
teacher to concretize the mathematical subject curriculum. The specificities of the criteria are described as 
follows: 

a) Technical usability 

• Ecise-of-use: SimReal+ should be easy to use, to start, and to exit. 

• Accessibility: SimReal+ should be accessible anytime and place. 

• Management facilities: SimReal+ should provide management facilities, e.g., it should be possible to 
store answers given by students. The tool should also have readily available content that can be modified. 

b) Pedagogical usability 

• Motivation : Motivation measures the extent to which SimReal+ is attractive to use, adapted to the 
students’ age, knowledge level, development, and interest, as well as tied to the students’ other activities 
and tasks. Using SimReal+ should be a motivational factor for learning algebra. 

• Variation: SimReal+ should be able to present the content in several ways, and facilitate various activities 
with students. SimReal+ should also be used as an alternative to achieve variation in teaching, and 
eventually in combination with textbooks and other study material and digital tools. 

• Student autonomy: SimReal+ should enable a high degree of autonomy so that the students do not 
necessarily need teacher assistance for help. The knowledge provided by SimReal+ should be potentially 
powerful to enable students to become less dependent on teacher assistance. 

• Individualization: SimReal+ should take into account adapted education, and different knowledge levels. 
The students should be able to work in their own pace, save their work and continue later. Disabled 
people should be able to work with the tool. 

• Differentiation: Differentiation means that SimReal+ should provide multiple tasks with different levels 
of difficulty, and can be tailored to the students. In addition, the tool should provide opportunities for the 
teacher to make individual adjustments and customize the tool when needed. 

c) Mathematical content 

• Quality of content: Mathematical content (theory, exercises and problems) provided by SimReal+ should 
be of high quality. The content should be mathematically sound and faithful to the underlying 
mathematical properties. SimReal+ should provide opportunities to display formulas correctly, and help 
students to gain knowledge that is otherwise difficult to acquire. 
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• Congruence with SimReal+ techniques : SimReal+ should enable the student to apply his or her own 
paper-and-pencil technique reasoning steps and strategies, and express mathematical ideas, as well as 
facilitate students’ mathematical activities. 

d) Assessment 

• Formative assessment is important to measure the extent to which SimReal+ provides several assessment 
and review modes, appropriate feedback in the process of problem solving, and use of several question 
types. Providing student profiles is important when it comes to adapt the questions to the student’s 
knowledge level. The quality of interaction and feedback to students’ actions are crucial for the learning 
process. In terms of summative assessment, the tool should provide teachers with quantitative data, 
statistics and results that help to evaluate students’ performances. It should give scores and grading. 

e) Teacher education 

• Adaptability and suitability of SimReal+ in teacher education includes the appropriateness of the tool in 
teaching school mathematics. This criterion also measures the extent to which SimReal+ provides 
opportunities to concretize the curriculum as specified by the New National Curriculum, and whether the 
tool is tied to teaching and adapted to the curriculum. 


4. RESEARCH DESIGN 


This research study involved 22 teacher students taking the course on digital tools in mathematics education 
in the fall semester of 2014. The students had very different knowledge background both in mathematics and 
digital tools. None of the students had any prior experience with SimReal+. The work used a survey 
questionnaire with open-ended questions to gather empirical data. To explore the use of SimReal+, teaching 
activities over a period of two weeks were designed and analyzed according to the purpose of this work. The 
activities were mostly related to the use of SimReal+ to teach trigonometry, but also other mathematical 
concepts. Table 1 gives an overview of the teaching activities over a period of three weeks. 


Teaching activity 

Lecture 

Lecture on 
mathematical tasks 
with SimReal+ 
Exercises using 
SimReal+ 

Lecture on SimReal+ 
and programming 


Tasks and exercises 
using SimReal+ 


Lecture on evaluation 
issues 


Table 1. Teaching activities 


Teaching aids 

• Video lecture 

• Teaching material 

• SimReal+ 

• SimReal+ 


• SimReal+ 

• ActionScript 

• SimReal+ 

• ActionScript 

• Diverse teaching 
material made online 

• SimReal+ 

• Survey questionnaire 


Subject 

SimReal+ User Interface 
2D calculator 
Introduction to SimReal+ 
Visualizations and simulations of 
trigonometric functions and concepts 
Simulations of trigonometric 
functions and other similar tasks 
Introduction to programming 
programming mathematical functions 
using ActionScript 

Students explore the program Moving 
a particle - Drawing a graph 
Students investigate the Pythagoras 
theorem, conic section, properties of 
triangles, cycloid, and 3D figures 
Evaluation of SimReal+ using a 
questionnaire with 73 statements 
grouped in 5 groups of criteria 


Date 

Week 

33 

Week 

34 

Week 

34 

Week 

35 


Week 

35 


Week 

36 


Duration 
45 min 

3 hours 

2 hours 

3 hours 

2 hours 
1-2 hours 


The teaching activities included video lectures, simulations of mathematical trigonometric functions, 
programming visualizations using ActionScript, and diverse teaching material, such as Pythagoras theorem, 
conic section, properties of triangles, cycloid, 3D figures, and similar mathematical functions. To measure 
the students’ responses, the survey questionnaire used a five-point Likert scale from 1 to 5, where 1 was 
coded as the highest and 5 as the lowest (l=“Strongly Agree”; 2 = “Agree”; 3 = “Neither Agree or Disagree”; 
4 = “Disagree”; 5= “Strongly Disagree”). The average score (MEAN) was calculated, and the responses to 
open-ended questions were analysed qualitatively. The survey included 73 items that were distributed as 
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follows: Technical usability (12 items), pedagogical usability (24 items), mathematical issues (16 items); 
assessment ( 12 items), and teacher education ( 1 1 items). The students were asked to comment each of the 
statements in their own words. In addition, the students were required to address 5 open-ended questions to 
express in their own words what they think on different issues of SimReal+. 


5. RESULTS 

The students’ responses to the survey questionnaire are described for the five broad categories that emerged 
from the evaluation criteria, and for supplementary issues that were addressed by open-ended questions. 

5.1 Global Impression of SimReal+ 

First, the students were asked to express their opinion on SimReal+, and what they globally liked and 
disliked about the tool. Basically, most respondents reported both positive and negative aspects of SimReal+: 
“7? seems like a good program, I like that it is free and easy to use. So you can use it when you want. 
Sometimes it can be a bit hard to find your way around”. Some students were more critical as the following 
comment indicates: “I think it is confusing. It has too many options that are poorly linked together. It could 
learn a lot from ’’Mastering physics”, in terms of more linear, logical structure and better feedback 
assessment. (...) I really like a teachers ’ opportunity to program (...) and add new graphics. ” 

5.2 Technical Usability 

Most students were globally satisfied with the technical usability in terms of easy-to-start, management 
facilities, availability of mathematical content and exercises, quality of video streaming and calculator, and 
accessibility of the tool anytime and anywhere as this students’ comment indicates: “It is not all that difficult 
for first timer to use SimReaI+ due to the information provided to make mathematics easy”. In contrast, 
students were not satisfied with the technical usability in terms of ease-of-use or user friendliness, 
navigability, response time, as this comment clearly reveals: “Everything does not work for me. I got 
confused. The long lists of lessons/exercises/ simulations in the rightmost window made it hard to focus 
attention to a specific task. Pre-made simulations and demonstrations worked well for illustration, but the 
visualizations that the activity should give, e.g. that Pythagoras ’ theorem is seen to be true, was not clear to 
me. The intention was clear, but I don ’t think the user will readily gain anything. ” 

5.3 Pedagogical Usability 

In terms of pedagogical usability, the students agreed that SimReal+ provides various mathematical activities 
and multiple representation of mathematical content, and that SimReal+ can be used as a lecture and textbook 
supplement. Concerning motivational issues, many students think that the tool is motivating to do 
mathematics with, as this comment shows: “Especially for mathematics interplay with physics, SimReal+ 
served as a good motivation and demonstration. However, motivation derives not only from an experience of 
«..., this is useful/beautiful», but also from one’s own development of capabilities to make mathematics 
useful/beautiful. This latter element of motivation was not offered by SimReaI+, so I only felt the brief sense 
of wonder, not of achievement and skillfulness ”. Furthermore, many students think that SimReal+ contains 
multiple levels of difficulty and provides opportunities for the teacher to make individual adjustments if 
necessary. However, the tool does not easily allow students to customize the tool. Nevertheless, the tool 
enables students to work at their own pace, which is a motivational factor in keeping students engaged in 
mathematics, as this comment reveals: “I think this is the greatest advantage of SimReal+ the opportunities 
for adjustment and individualization. The teacher knowing a little programming (as should all upper 
secondary teachers in my opinion ) can rather easily make small addition and alterations to a 
pre-programmed lesson, so it fits his needs: add a second particle, hiding/showing the kinematic (...) 
equations, etc. ”. Moreover, most students think that SimReal+ does not fully allow to work independently 
from teacher assistance or fellow students. Also textbooks are still needed when using SimReal+ in 
classroom: “Autonomous learning is not well-supported, only autonomous playing around without goal. And 
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for the purpose of playing with mathematics, there were too few options for the student, (....), and mostly 
teacher-chosen and implemented possibilities/options’’. Furthermore, most students agreed that SimReal+ is 
not fully appropriate to use as an alternative to achieve variation in teaching mathematics as this comment 
clearly shows: “ The problem as I see it is that there is no clear-cut connection between the variations of a 
specific topic or notion. For example, changing the parameters in a wave function, (...) and its exact 
correlation with the graph, (....), but such things are not in any way implemented in SimReal+. " 

5.4 Assessment Issues 

Most students indicated that SimReal+ does neither provide a diagnosis of student’s problem solving nor 
appropriate feedback that is adapted to the students’ knowledge level. In addition, SimReal+ does neither 
build student profiles nor serve up appropriate questions or several questions to the students. Furthermore, 
SimReal+ does neither have a review mode showing what the student has done wrong or right, nor allow for 
the use of several question types. The lack of student profiles may prevent students from engaging in 
authentic learning activities with SimReal+. Hence, it is obvious that the role of the teacher is still important 
in assessing the students’ learning of mathematics. The following comment summarizes the limitations of 
SimReal+ in terms of assessment: “SimReal+ is very weak with respect to feedback. I experience no 
feedback or assessment of any work or exercises. ” 

5.5 Mathematical Issues 

More than the majority of the students believed that SimReal+ has a good quality of mathematical content in 
terms of mathematical correctness and representation of mathematical properties and operations, e.g., 
formulas, functions, graphs, and geometrical figures. A typical comment was: “SimReal+ is rich in 
mathematical content. It gives systematic procedure for solving a particular problem in mathematics. It is 
able to translate algebraic expressions into graphs or geometric figures, etc. The video and audio visual 
aspect of the software enhances deeper understanding of mathematical concepts’’. However, the practical 
applications and exercises are not fully well-designed as this comment indicates: “ The exercises were too 
general, not specifically focusing on some mathematical learning object. For example, the most focused 
exercise was about changing angles into radians and vice versa. The connection was very well visualized on 
the unit circle, but there weren’t added any techniques to do the calculations. (...) The exercises were 
well-formulated but they were too focused on visualizations done by the program with no connection to 
paper-pencil work or text book theory”. Furthermore, SimReal+ is useful when it combines video lessons, 
simulations, live streaming of lessons, and exercises. It helps to acquire new mathematical knowledge, and 
in a lesser degree, it is congruent with paper-pencil techniques. SimReal+ also provides opportunity to help 
students gain knowledge that is otherwise difficult to acquire. In addition, the tool’s openness enables 
students to express mathematical ideas. The negative side of SimReal+ is that students do not like 
programming mathematical visualizations. They prefer focusing on the mathematical part of the program, 
which is more important than understanding programs written in Action Script as this comment expresses: 
“Maybe for the students, but they don '( need all the programming stuff, just an interface where mathematical 
properties can be approached. (...) For the teacher, I think it is essential that he/she understands the 
programming enough to make non-trivial changes such as adding (...) a button with a new function. ” 

5.6 Adaptability of SimReal+ to Teacher Education 

Most students think that SimReal+ could be an appropriate tool in secondary schools, but not in middle or 
primary schools. SimReal+ also enables teachers to concretize the curriculum. However, when asked 
whether they will continue using SimReal+ for teaching mathematics, most of them answered negatively as 
this comment clearly reveals: “I will use SimReal+ to further develop an illustration of a phenomenon in my 
teaching but only for illustration, not interactive student activity. ’’ 

5.7 Flipped Classroom 

The students were asked whether flipped classroom will be of interest and whether new digital tools can give 
students new possibilities in teaching mathematics. Most students agreed that flipped classroom is of huge 
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interest, but “(...) we need to focus and stop believing that just planning around will result in learning. We 
need strictly directed lessons that allow for continuous interchanging of visual illustrations, theoretical 
presentations, exercise solving, etc. In short, digital tools could help us in writing the classroom activities 
that have traditionally been separated in time and space. (...) Flipped classroom also is very interesting but 
needs highly motivated students. 

5.8 Programming Visualizations 

The students were asked in what way do they think programming mathematical visualizations by their own 
will help them in understanding mathematics if they can use different templates so the concentration can be 
on mathematics and not on difficult details in the programming process. As mentioned earlier, students prefer 
focusing on the mathematical part of the program rather than programming. Most students agreed that 
templates can help to concentrate on the mathematical part, but I need very specific templates focusing 

on illustrating a specific notion, e.g. the kinematic equations of cycloid (....), so I need to know in advance 
rather precisely which properties lam investigating. (...). The developer of SimReal+ is planning to design 
such templates so that students can be able by their own to program mathematics (elementary and advanced 
visualizations and simulations) directly into their own Web pages without any special tool. They just need to 
write their own mathematical code. Most students like the idea of using templates, as this comment clearly 
highlights: “I think it is great. I think there should be a clear distinction in the programming interface 
whether you are implementing graphics mathematical properties, exercises and theory (....). Then you can 
access more readily different settings that a visualization or simulation should offer to a student using it. ” 

5.9 Suggestions for Improvements 

Finally, the students were asked to provide suggestions to make SimReal+ more appropriate for use in 
teacher education. Perhaps, the most interesting suggestion is expressed by the following comment: ‘Focus. 
It tries to do everything, but really only presents a fragmentary list. Technologically, it needs a more directed 
structure such that one task leads to the next in a logical way, not just present an array of tasks. (...) 
Mathematically, it must connect the different settings graphical, algebraic, tabulator, arithmetic, geometric, 
etc. Assessment-wise, it must offer feedback to the implemented exercises. ” 


6. CONCLUSION AND RECOMMENDATIONS 

The number of participants ( N=22 ) may not be sufficient to adequately support the generalization of the 
results. Hence, new cycles of experimentations and evaluations of SimReal+ are warranted to generalize the 
findings of the present work to ensure more validity and reliability. However, despite the limitations of the 
study, it has been possible to make some reasonable interpretations of the results and draw some 
recommendations for using SimReal+ in teacher education. Firstly, the use of SimReal+ indicates that the 
tool shows potential for teaching mathematics that is suited to the students’ knowledge level, although not all 
criteria are equally met. Secondly, SimReal+ is technically well designed in terms of accessibility and 
management facilities. Furthermore, SimReal+ covers a wide range of mathematical content with varied 
levels of difficulty, including school mathematics. Likewise, the content is mathematically correct and 
reflects the underlying properties of mathematics at different levels. However, mathematical and technical 
issues in themselves are not sufficient to make SimReal+ become an integral tool in learning and teaching 
mathematics in a school environment. SimReal+ is not pedagogically well designed to ensure a smooth 
integration of the tool in teacher education in terms of variation, student autonomy, differentiation, and 
individualisation, and assessment issues as well. Hence, SimReal+ in its present form is not fully appropriate 
for use in teacher education, unless didactical functionalities and pedagogical modalities of using SimReal+ 
are considered in future versions of the tool. Likewise, the user interface must be simplified to make 
SimReal+ more intuitive and easy to use in teacher education. Summarizing, the use of SimReal+ in teacher 
education needs to take into account the pedagogical dimension of teaching and learning mathematics in 
order to adapt the tool to the modalities of using digital tools in teacher education. 
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ABSTRACT 

Improving student retention is an important and challenging problem for universities. This paper reports on the 
development of a student attrition model for predicting which first year students are most at-risk of leaving at various 
points in time during their first semester of study. The objective of developing such a model is to assist universities by 
proactively supporting and retaining these students as their situations and risk change over time. The study evaluated 
different models for predicting student attrition at four different time periods throughout a semester study period: 
pre-enrolment, enrolment, in-semester and end-of-semester models. A dataset of 23,291 students who enrolled in their 
first semester between 2011-2013 was extracted from various data sources. Three supervised machine learning 
techniques were tested to develop the predictive models: logistic regression, decision trees and random forests. The 
performance of these models were evaluated using the precision and recall metrics. The model achieved the best 
performance and user utility using logistic regression (67% precision, 29% recall). A web application was developed for 
users to visualise and interact with the model results to assist in the targeting of student intervention responses and 
programs. 


KEYWORDS 

Attrition, retention, student, prediction, machine learning. 


1. INTRODUCTION 

The student retention rate is a measure of student success used by many universities. Measuring student 
retention is important because it can reveal how well a university is able to retain students based on the 
quality of education, research and services provided (Kim 1998). Improving student retention is a highly 
desirable and challenging goal for universities as a declining student population can significantly affect 
current and future students, instructors, researchers, professional staff and the university as an institution 
(Ifenthaler and Widanapathirana 2014. Kovacic 2012, Murtaugh et al. 1999, Olani 2009). 

There has been a long history of research into studying and building models concerning student retention 
and attrition (Braxton 2000. Reason 2009). The most significant benefit of developing such models is to 
enable student support staff to proactively conduct interventions to assist and retain at-risk students. This can 
result in other benefits such as improved student graduation rates, better support for students and increased 
university revenue depending on the quality of the predictive model and student interventions. 

There are two primary challenges involved in developing a first year student attrition predictive model. 
First, predictive models need to be developed under rigorous experimental settings for training, validating 
and testing the model’s prediction performance on historical student data. Second, if the model is used to 
generate predictions at different points in time during a student’s time at university, the model can only be 
trained with data available at these respective points in time. For example, a model predicting a student’s risk 
of attrition at the beginning of their first semester cannot be trained on data that is only available after they 
have completed the semester. The risk of attrition is defined for this research study as the likelihood of a 
student leaving the university in the following year. 
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This paper reports on several research contributions. First, a taxonomy of common student attrition and 
retention features is synthesised from the literature. Second, the problem definition describes different ways 
student attrition can be defined and formally defines the problem of predicting student attrition in this paper. 
Third, a student attrition model is described that contains features from four different time periods through a 
university student’s first semester of study. Fourth, the settings of the experiment in terms of workflow, 
machine learning methods and performance metrics used to develop and validate the predictive model are 
outlined. Last, the results from our experiments are presented and the insights gained from the research are 
discussed. 


2. TAXONOMY 

Student attrition and retention research literature was reviewed and a taxonomy of common student attrition 
features was developed (Table 1). Many of these research papers evaluate features from multiple categories. 
Demographic (Bogard et al. 2011, Moller-Wong and Eide 1997, Yu et al. 2010) and academic history 
(Kovacic 2012, Luna 2000, Olani 2009) were the most common studied features, in particular age, gender, 
ethnicity and high school grades were evaluated by the majority of the papers. Social (Dey and Astin 1993, 
Kim 1998), psychological (McKenzie and Schweitzer 2001, Olani 2009) and financial (Dey and Astin 1993, 
Reason 2009) features were less common. 

The university category represents features evaluated after the student has enrolled and commenced 
studying. This category comprises of six subgroups containing enrolment (Bogard et al. 2011, Fike and Fike 
2008), course (Lin et al. 2012, Murtaugh et al. 1999), units (Dekker et al. 2009, Kim 1998), resource use 
(Balakrishnan and Coetzee 2013, Dietz-Uhler and Hurn 2013), engagement (Glogowska et al. 2007, 
Murtaugh et al. 1999, Xuereb 2014) and during study features (Bogard et al. 2011, Dekker et al. 2009, 
Kovacic 2012). Measuring the usage of university resources such as the LMS, library and video lectures is 
less common in the literature but will likely receive more attention due to the popularity of delivering digital 
education and Massive Open Online Courses (MOOCs) in recent years. 


Table 1. Taxonomy of Student Attrition Features 


Category 

Examples 

# Sources 

Demographic 

Age, gender, ethnicity, living on campus, has children, parents’ 
education, has disability, experience working in study area 
Hours spent socialising with friends, exercising, engaged in 
sporting teams, volunteer work, hobbies 

Personality traits, self-esteem, commitment to chosen profession, 
informal / formal support, GRIT (12 question survey) 

Household income, requires financial aid, currently working 
High school grades, high school rank, public vs. private high 
school, other qualifications (previous diplomas, degrees) 

11 

Social 

2 

4 

6 

12 

Psychological 

Financial 
Academic history 

University 


- Enrolment 

Enrolled in first preference course, enrolled before course start 
date, language tests taken, reason for attending university 

5 

- Course 

Difficulty (pass rate), student has chosen a major / specialization, 
attendance mode, load type, field of study 

7 

- Units 

Number of units enrolled, number of contact hours, average pass 
rate, unit type, quality of lecturer, quality of lecturer feedback 

4 

- Resource use 

Library, learning management system, lectures e-mail 

2 

- Engagement 

Student survey participation, student clubs / groups membership, 
enrolled in educational support programs 

3 

- During study 

Current course average, number of units the student is currently 
failing, attended hours, units passed, units failed 

5 
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3. PROBLEM DEFINITION 

There are many variations in defining the risk of student attrition at universities determined by stakeholder 
perspectives and needs and the time period for the desired prediction (Table 2). For example, a predictive 
model could be developed for a unit coordinator to identify students who are at-risk of withdrawing from 
their units this year. Alternatively, another model could be developed at the faculty level to identify students 
at-risk of changing courses outside the faculty in the next semester. 


Table 2. Student Attrition Definition Factors 


Stakeholder 

Enrolment 

Desired prediction time period 

Unit coordinator 

Unit 

This semester 

School 

Course 

Next semester 

Faculty 

University 

Next year, 2 years, 3 years etc. 

Institution 


Student life-time 


The problem of predicting student attrition is formally defined as a binary classification problem. The 
dataset can be described as a set of students S = (s 1; s 2 , ... , Sj, ...,s n ] and an attrition label Y = (y 1 = 
retained, y 2 = attrition } where s t represents the i-th student in S. Students are represented as a set of 
student attrition features (factors) x l in the model as defined for Sj. 

p y f — f yl yl yf y 1 "1 

A, 2 » ■■■ » A 'j t ■■■ > A, mS 

tfi (x‘, y k ) is a Boolean function that is used to determine whether s t belongs to y k where k = {1, 2}. 

>p( xi ,y k ) : X x Y -» {True, False] 

This definition can be applied to predict student for different factor mappings (Table 2). The focus of this 
research is to develop a model for predicting student attrition for the institution stakeholder, at the university 
enrolment level and the desired prediction time of next year. 


4. STUDENT ATTRITION MODEL 

A conceptual model for predicting student attrition was developed after reviewing student retention / attrition 
research papers in the literature, discussions with domain experts and academics, identifying available 
student data from university data sources in addition to proposing new features based on our knowledge 
gained through this process. The feature types used in the conceptual model in relation to the proposed 
taxonomy (Table 1) are demographic, academic history, enrolment, course, resource use, engagement and 
during study features. 

An initial set of 28 features are evaluated in a set of experiments based on the feasibility of extraction and 
data handling during the time available for the research. The data system of the university is being built for 
future automated extractions that will utilise these research findings and will allow more features to be 
included in the model. Due to these considerations, the research scope is restricted to only evaluating student 
data in their first semester of study. 

However, a student’s risk of attrition is likely to change over time as more data becomes available on how 
the student is progressing throughout the semester. Therefore, the study evaluates four models using features 
that are available at four different time periods in the semester. These models are the pre-enrolment model, 
enrolment model, in-semester model and end of semester model (Table 3). 

The pre-enrolment model includes features that can be used to predict a student’s risk of attrition before 
they commence studying at university. This model contains 17 features based on a student’s demographics, 
academic performance at high school as well as whether a student has participated in the university enabling 
program or has taken bridging units at the university college as an alternative pathway of entry. 

The enrolment model contains an additional 5 features based on data collected from the student enrolment 
process such as their age at enrolment, the workload of the course (course credit value), whether they are 
studying on campus or externally (attendance mode), their field of study and whether they are studying 
full-time or part-time (study load type). This model can be used to predict a student’s risk of attrition after a 
student has completed their enrolment at university. 
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Table 3. Student Attrition Model Features 


# 

Pre-enrolment 

# 

Enrolment 

1 

Gender 

18 

Age at enrolment 

2 

Birth country = Australia 

19 

Course credit value 

3 

Birth country region 

20 

Attendance mode 

4 

Home language = English 

21 

Field of study 

5 

Aboriginal or Torres Strait Islander 

22 

Study load type 

6 

Citizenship 



7 

Has disability 



8 

Western Australia metropolitan status 

# 

In-Semester 

9 

Socio economic status 

23 

LMS logins 

10 

High school / tertiary entrance score 

24 

Portal logins 

11 

Course preference number 



12 

High school type 



13 

Highest education qualification 

# 

End-of-Semester 

14 

Elite athlete 

25 

Course average 

15 

University enabling program participant 

26 

Surveys completed 

16 

Completed university college program 

27 

Units completed 

17 

First in family 

28 

Units withdrawn 


The in-semester model contains an additional 2 features that measure a student’s usage (i.e. logins) of the 
university’s Learning Management System (LMS) for accessing course materials and submitting assessments 
as well as the university’s student portal for accessing official communications and other university student 
Web applications. This model can be used to predict a student’s risk of attrition at different points in time 
during the semester. 

The end-of-semester model contains an additional 4 features that measure the student’s academic 
performance at university in terms of their end of semester grade (course average), whether the student 
participated in teacher and unit feedback surveys in addition to the number of units completed and withdrawn 
for the semester. This model can be used to predict a student’s risk of attrition after they have completed their 
first semester. 


5. EXPERIMENTAL SETTING 

The experiment comprises of nine main tasks (Figure 1): feature extraction, feature normalisation, feature 
encoding, cross validation, feature selection, model training and tuning, prediction, performance evaluation 
and error analysis. These tasks will now be described in detail. 

The dataset is comprised of 23,291 students who commenced their first semester at Curtin University, 
Australia between 1 Jan 201 1 and 31 Dec 2013. This data was extracted from various university systems and 
data sources such as student administration, business intelligence, LMS, student portal, unit and teacher 
feedback surveys, university enabling programs and college student records in addition to external sources 
containing student tertiary entrance scores and preferences as well as Australian census data. After data 
cleansing and preparation, the dataset consists of 19,222 (83%) retained and 4,068 (17%) attrition students. 

Feature extraction acquires the data required for the model features in addition to calculating the student 
attrition labels (based on definition in Section 3) used for training, validating and testing by the predictive 
models. Feature normalisation refers to mapping feature values into a new numerical range such that the 
relative magnitude is preserved. Normalisation is important because it avoids modelling features in larger 
numeric ranges that can dominate features in smaller ranges. It is also more computationally and memory 
efficient to perform computations on smaller scaled values than larger data values (Tax and Duin 2002). The 
features are normalised to a scale of [0, 1] in the experiments. 

Data discretisation involves portioning feature values into ranked intervals or bins. Each interval is then 
treated as a categorical value. Discretisation has been shown to improve classification performance 
(Dougherty et al. 1995, Fayyad and Irani 1993, Kotsiantis and Kanellopoulos 2006, Liu 2007). The resulting 
full model comprises of 146 encoded features selected after experimenting with various feature discretisation 
and encodings to optimise prediction performance. 
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Figure I . Experimental workflow 

A stratified 10-fold repeated random sub-sampling validation cross-validation technique (Liu 2007) is 
applied for this project. Students are randomly assigned into a training (60%), validation (20%) or test (20%) 
dataset. This assignment is performed 10 times in a stratified manner to generate 10 different training, 
validation and test datasets. Stratified means that the class distribution of retained (83%) and attrition (17%) 
students are preserved in each dataset. Therefore, 10 different predictive models are built for each dataset and 
their results are averaged to measure overall performance. 

Feature selection is the process of selecting a subset of the most important features for use in model 
construction. Recursive Feature Elimination with Cross- Validated (RFECV) selection is employed to identify 
the best feature subset in addition to applying LI regularisation (Lee et al. 2006) for logistic regression. 
Decision tree and random forest employ embedded feature selection by design as only the most important 
features are used to construct the tree(s). 

With respect to predictions (Kotsiantis and Kanellopoulos 2006) reviewed a number of candidate 
supervised machine learning techniques and their suitability based on different factors. The aim of the student 
attrition model is to achieve strong prediction performance, where insights can be understood and assist 
student support staff in prioritizing the various risks of attrition within their workload of tailoring student 
interventions. Based on this aim, the model employed three supervised learning techniques: LI regularised 
logistic regression (Cox 1958, Lee et al. 2006, Walker and Duncan 1967), decision trees using the CART 
algorithm (Breiman et al. 1984) with the Gini impurity criterion and random forests (Breiman 2001). 

Precision and recall metrics are used to evaluate the performance of the predictive models. Precision 
measures the percentage of students the model correctly predicted as attrition. Recall calculates the 
percentage of at-risk of attrition students correctly identified from the test dataset. These metrics are more 
effective at assessing performance on imbalanced datasets than alternative metrics such as classification 
accuracy and receiver operating characteristic (Davis and Goadrich 2006). 

Error analysis is an iterative process of evaluating the model errors to gain insights in further improving 
the model. Learning and validation curves are constructed to evaluate the model on the training and 
validation datasets. These curves allow for the identification of model bias and variance in addition to 
evaluating the effectiveness of model tuning tasks. A web application was developed to visualise and interact 
with the model results. 

6. RESULTS 

The end-of-semester model comprises of 28 features transformed into 148 encoded features. Furthermore, the 
in-semester model is split into four additional time periods from weeks 1-4, 1-8, 1-12 and 1-17. The purpose 
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for experimenting with these time periods is to determine how model performance changes as more student 
data is collected throughout the semester study period. 

The experimental results for the time based model and each machine learning classifier are presented 
(Figure 2). Random forests achieved the top precision (0.30-0.71) after the in-semester weeks 1 to 4 period 
and intermediate recall performance (0.08-0.33). Logistic regression achieved the top precision (0.46-0.67) 
for the pre-enrolment and enrolment time periods and the lowest overall recall performance (0.01-0.29). 
Decision trees achieved the top overall recall [0.08, 0.37] but lowest precision performance [0.27, 0.60] 

Logistic Regression '■"Decision Tree “♦■Random Forest 



Figure 2. Experimental Results 

The experiments show that the end of semester model achieved the best performance, underscoring the 
expectation that classifiers should achieve better performance as more data is obtained (Figure 2). In terms of 
overall performance, decision trees achieved the best recall but logistic regression and random forests 
achieved better precision. Logistic regression outperformed random forests in terms of precision for the 
pre-enrolment and enrolment models but achieved relatively on -par performance with the in-semester weeks 
1 to 4 models, which suggests that that logistic regression might be better suited for predicting student 
attrition for these semester time periods. Conversely, random forests outperform logistic regression for the 
remainder of the semester. In practice, one would likely decide upon using one of these models based on 
other non-performance factors as they have achieved relatively similar precision results. 

The model features have different values of importance and strength for each of the time period models 
and machine learning methods. Overall, the most important features that significantly improved prediction 
performance are; the course average, LMS logins, portal logins, study load type and course credit value. The 
other model features make small contributions that are significant when added together. The strength 
(coefficients) of the encoded features are not presented due to length constraints. 

The recursive feature elimination and cross validation feature selection method was used in parallel to 
training the logistic regression model with the full feature set. In our experiments, the feature selection model 
often achieved on par and improved performance compared to using the full feature model on the validation 
datasets. However, this result was not reflected when the models were evaluated on the unseen test datasets 
as the feature selection models achieved overall lower performance to the full feature model. The training 
and validation of the full feature model is not computationally intensive so it is used rather than the feature 
selection model for generating predictions on current Curtin University students. 

The two best performing classifiers in terms of precision are logistic regression and random forests. 
However, it is easier to extract useful insights and examine the logistic regression model than random forests. 
We can evaluate the coefficients of each feature and identify the top scoring attrition and retention features 
for individual students with logistic regression. For example, one student may have their low course average 
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score as their highest contributing feature to attrition while another student with a higher course average may 
have their age at enrolment group (e.g. 40-49) as their highest risk of attrition feature. 

Random forests involve building a number of decision trees (10 in our experiments) and averaging the 
predictions from these trees. Therefore, it is more difficult to explain and visualise how predictions are made 
from 10 different decision trees. Therefore, logistic regression is recommended over random forests as it 
achieved similar precision performance but is easier to understand, extract insights and plan student 
interventions. A web application was developed to visualise the logistic regression model has been made 
available to university student support staff (Figure 3). 

Student Attrition Predictive Model 

Home > Predictions > Student 

Demographic 

• Male 

• 27 years old at enrolment 

• Non- Indigenous 

• Citizenship - Domestic 

• Birth Country - Australia 

• WA Metro Status - Metropolitan 

• Permenant address category - Urban 

• Sock) economic category - High SES 

• Elite atheiete - No 

• Disability - No Disability 

• First in family - No (PG) 


Risk of Attrition: 54.8% 


7. CONCLUSION 

This study developed a student attrition model that predicts a student’s risk of attrition at different time 
periods in their first semester: the pre-enrolment, enrolment, in-semester and end-of-semester periods. The 
model performance was validated on university students who enrolled in 2011-2013. Three supervised 
learning methods were evaluated; logistic regression, decision trees and random forests. Logistic regression 
and random forests achieved the best precision performance on this dataset and based on considerations of 
ease of use and fitness for purpose, a web-based interface was constructed using the logistic regression 
model, due to its relative ease in extracting insights to assist university staff in understanding and planning 
student retention interventions. Future work will involve experimenting with additional features such as 
student assessment data from the university’s LMS to further improve the model and to build a dedicated 
data exploration environment to make university data more readily available for building, refining and 
validating predictive models. Next steps also include reviewing and understanding the implications of the 
predictive methodology on retention policy, practices and technologies in a university span with a wide range 
of potential changes to business practices, policy concerns and practical implementation issues. 


Top 5 Selected Attrition Factors Top 5 Selected Retained Factors 



Academic history 

• Highest education Qualification - Not Entered 

• ATAR score - 69 9999 

• TISC preference - 1 

• High school type - Non-Government 

• Uniready -No 

• Curtin college - No 

Enrolment 

• Course • B COMM • Bachelor of Commerce 

• Course credit value - 600 

• Enrolment year -2015 

• Attendance mode - External 

• Study load - Part-Time 

• Faculty Curtin Business School (CBS) 

• School - Curtin Business School (CBS) 


■ Atar score band; 70.00 to 74.95 

■ Age group at enrolment: 25-29 

■ Course credit value: 600.0 

■ Birth country un region name: Oceania 

■ Socio economic category: High SES 


■ Field of study: Management and Commerce 

■ Educational sector: Non-Government 

■ Tisc preference number: 1.0 

■ Wa metro status: Metropolitan 

■ Highest education qual: Not Entered 


Figure 3. Interactive Model Results Visualisation 
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ABSTRACT 

This study examines the roles of Canadian school leaders in response to the rising phenomenon of student use of social 
media which impacts school climate and safety. The use of social media has resulted in more online text and image -based 
communication to multiple users and less face-to-face communication with single users. Adolescent communication, a 
previously invigilated phenomenon, has not yet been replaced by an online social presence with a social regulation. 
Secondly, there have been changes in national, provincial, and district Safe School policies in response to online 
misbehaviour that impacts student safety within the school environment. This small study considers the views of nine 
Canadian secondary school vice-principals about school policies and students’ cyber behaviours. Their responses were 
collected on a NING, a private cyber environment. Findings indicate that when cyber events come to the awareness of the 
school administration, the school becomes a nexus for investigation and resolution. The study also finds that when 
Canadian secondary school administrators are compelled to respond to the event, these school leaders can and do access 
social media, employ cyber skills to identify users, intervene in wrongdoing and, in the process, follow and enact 
Canadian Safe School legislation. However, these same school leaders express great reluctance to use social media for 
their personal or professional purposes. The authors hypothesize that this may be due to their exposure to negative 
experiences with social media in schools. 


KEYWORDS 

Safe School policy, cyber behaviour, social media 


1. INTRODUCTION 

Online correspondence occurs at a distance - allowing a sender to be geographically and physically located a 
distance away from the receiver. With the introduction of handheld devices, users may also be young adults, 
adolescents, or children sending messages to multiple people without the awareness or consent of a 
supervising adult. Both social presence and regulation are impacted. Unfortunately, these online activities 
may involve a range of cyber-misbehaviours which can result in loss of privacy, loss of identity, and 
cyber-bullying. Cyberbullying is an international phenomenon that is increasing, and teachers are not able to 
respond or feel they are adequately prepared to deal with these events, particularly since they often occur 
outside of the school day (McNamara & Moynihan, 2010). 

In Canada, Safe School policies which are designed to respond to issues such as bullying and 
cyberbullying, fall under the purview of each of the thirteen provinces and territories. One of these provinces, 
Ontario, has recently passed legislation: Bill 13, the Accepting Schools Act (Legislative Assembly of Ontario, 
2012) which enables administrators to investigate and respond to these online contraventions of Safe School 
policy. This legislation requires schools and specifically school administrators, to address harassment, 
bullying, and discrimination in all forms, including online misbehaviour which occurs outside of school 
hours but impacts student-to-student relationships. 

This study uses qualitative methodology to examine secondary school vice -principals’ responses to 
incidents of cyberbullying over a six-week period using a private social media network, examining the roles 
of social presence and restorative practices in their responses to incidents of cyber aggression and online 
bullying. Social presence, earlier theorized in the context of building community in online teaching (Aragon. 
2003) theorizes the separation or isolation of the online user. Rettie (2003) sees social presence as the judging 
of the perception of the other online participant. Schools are sites where cyber behaviour is investigated and 
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attempts are made to resolve the conflicts. Little is known about the lack of social presence in 
cyber-behaviours and its effect on school climate. This study expands our understanding of social presence 
where its absence may contribute to cyberbullying and its presence may be seen as a means to redress the 
wrongs. 

Findings from this study indicate that school leaders recognize that school is a connection space for 
students, and they feel a strong sense of obligation to keep students safe, including during events that occur 
online and out-of-school. One unintended repercussion of the school leaders’ online investigations, however, 
is that they tend to avoid social media for their personal and professional use due to its negative associations. 


2. RESEARCH ON SCHOOLS AND CYBERBULLYING 

The rise of online communication has implications for schools such as cyberbullying and its impact on school 
climate. Another area it impacts is policy. There are multiple policy responses to be considered in the 
Ontario, Canada legislative context. Theory related to social presence and social regulation also contributes 
to the deliberations and discussion in this paper. Although one research study cannot address every 
implication of social media use in schools, the authors endeavour to shed a light on the specific insights of a 
small group of Canadian school leaders who are working with new policies to help students be more aware of 
the impact of their actions on themselves and others. 

2.1 Schools and Cyberbullying 

Adolescents have access to new communication technologies all of which can establish instantaneous and 
anonymous communication, and are increasingly simple to operate. Simultaneously, there has been a rise in 
bullying and cyberbullying which now impact adolescents and this type of online aggression is perceived to 
be increasing (Calhoun & Daniels, 2008; McNamara & Moynihan, 2010). The examination of this issue has 
been identified as a “relatively new and emerging field of research” (Cassidy, Faucher, & Jackson, 2013, p. 
576). Research from multiple countries indicates that cyberbullying is an international phenomenon and that 
schools are seen to bear some responsibility for responding to bullying and online aggression (Boulton & 
Mirsky, 2006; Calhoun & Daniels, 2008; Grossi & dos Santos, 2012; Flasenstab, 2012; Mark & Ratliffe, 
2011; McNamara & Moynihan, 2010). 

Education law and policies also recognize that schools have a role in responding to negative cyber 
phenomena. Ontario, Canada has published a series of policy responses in the past fifteen years toward 
promoting Safe School environments. Bill 81, (Legislative Assembly of Ontario (LAO), 2000) not only gave 
teachers the authority to suspend students, it required teachers to do so and gave teachers the right to extend 
suspensions issued by the principal. This legislation was modified in 2007 with Bill 212 through an 
amendment to the Education Act with respect to behavior, discipline and safety. Bill 212 required offenses 
of bullying to be mandatory suspensions of up to 20 days but also allowed school leaders to consider 
mitigating factors (LAO, 2007, S306 (3)). 

Most recently. Bill 13, the Accepting Schools Act, 2012 (LAO, 2012) amends the Education Act, 
separating definitions of bullying and cyberbullying and holding schools accountable for dealing with issues 
surrounding both offenses. This Act also requires schools and districts to create and enforce equity and 
inclusive education policies, and bullying prevention and intervention plans. School leaders must take active 
measures to ensure a positive school climate. This Act also adds more protection for victims’ rights and 
stricter protocols for confidentiality in light of increased occurrences of online conflict (LAO, 2012). 

In summary, school administrators have a legislated responsibility to maintain a Safe School environment 
and respond to incidents of harassment and discrimination if they impact the safe school culture. In both 
policy and practical terms, school is a nexus where the implications of 24/7 cyber events are manifest. School 
leaders are also ideally situated to access online communications to investigate incidents and to meet with 
students and their parents in a setting familiar to both. For these multiple reasons, schools are considered to 
be well-situated to respond to bullying and online aggression. 
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2.2 Social Presence Theory 

Technologies such as the telephone started to change aspects of face-to-face communication. Short, Williams 
and Christie (1976) defined social presence at that time as the “degree of salience of the other person... and 
the consequent salience of their interpersonal interactions” (p.l). When Korzenny (1978) observed that 
face-to-face interactions were being replaced by electronic devices which altered long-established social 
patterns, he coined the term “electronic propinquity” to describe the changes in proximity when experiences 
became less direct than in person. Walther ( 1992) examined early studies on changes in the relational aspects 
of computer-mediated communication. He found that social connectedness took more time in the 
computer-mediated settings; the degree of connection was affected by the type of technology in use. 
Korzenny (1978) and Walther ( 1992) pioneered theory in social presence. 

A key consideration in online learning theory is building community for distance learning (Aragon, 2003; 
Garrison, Anderson & Archer, 2000; Hege; 2011; Korzenny, 1978; Rettie, 2003). Definitions of social 
presence usually include some reference to the subjective perception of participants to other participants, 
sometimes in the context of learning communities. However, little research makes connections among social 
presence, social media, and school-based responses to cyber-behaviours. We hypothesize that, through 
legislation and predisposition, schools offer spaces where community and social presence can be re-built 
using restorative practices. 

Aragon (2003) theorizes that interpersonal contact is more difficult to establish in the online setting 
because it affects proximity, eye contact, body language, and interpersonal awareness. Rettie (2003) sees 
social presence as a judgement of how the other participant perceives the communication or the event, 
including cognizance of the other person and their intention. 

Deng & Tavares (2013) link social presence to student motivation, finding that students report feeling 
more at ease and casual on Facebook, with more connectedness and belonging. Hege (2011) cautions 
instructors to have a second Facebook account for professional purposes only. Facebook may increase social 
presence through immediate feedback and its “likes” function, while Moodle seems more formal, with less 
regulating instructor influence (Deng & Tavares, 2013). Social presence has also been described as the 
ability of online users to project their individual (personal) characteristics into the online community “real 
people” (Garrison et al., 2000, p. 94). Multiple researchers (Walther, & Bazarova 2008; Yamada & Goda, 
2012) identify that social presence adds subjective qualities to online environments such as support for 
co-operative learning (Oyarzun & Morrison, 2013), building enjoyment, and virtual roles (Yeh, Lin & Yu, 
2011). 

Little attention to date has focused on the impact of the lack of social presence or gap in time to build 
social presence which may be occurring in cyberbullying and online conflict in schools. More studies are 
needed to understand how individuals operate within the anonymity of the online world. We theorize that a 
lack of social presence or time to develop social presence contributes to the negative quality of peer-to-peer 
digital communications, which can be corrected through person-to-person resolutions. 

This research study examines the re-creation of social presence and regulation and the roles that 
administrators can play to help students understand relational issues created by or impacted by cyberbullying 
and online conflicts. This particular paper focuses on aspects of the study which reveal how secondary 
school administrators use social media to investigate and respond to online conflicts and cyberbullying, and 
how, in this process, they help increase social presence and social regulation. This study also investigates 
how administrators use social media for professional purposes such as investigating and documenting events 
of cyber harm. 


3. METHODOLOGY 

This study employs essentially qualitative methodology, including an online survey for the participants and 
collection of their perceptions about cyber incidents through a NING, which is a private blogging site. The 
Limesurvey was chosen as it is considered to be a trustworthy source of the anonymous data collection 
(Klieve, Beamish, Bryer. Rebollo, Perrett, & Van Den Muyzenberg, 2010). The quantitative questions in the 
survey included questions based on gender, age, years of experience, understanding of policy legislation, and 
familiarity with technology. 
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3.1 NING as Repository 

An online private social network (NING) was used to collect the qualitative research. The NING requires a 
private login and password. The NING acted as a repository of information that captured the story-like 
experiences of the vice-principals and, it was hoped, would foster online discussion among the vice 
principals who could access it 24/7. Questions were asked that were intended to elicit their knowledge of and 
response to online aggression and cyberbullying as well as their understanding of relevant legislation in the 
context of their roles. Of added research value were the revelations the vice-principals made in relation to 
their use of social media both in schools and outside of them. 

3.2 Access and Permissions 

Three district school boards were approached for consent, and only one board chose to participate. 
Significantly, one school board felt that the research project relied too heavily on the use of an open source, 
free subscription type application to gather data. This school board cited the protection of the gathered 
information as a concern as well as information about events that happen at the school and the potential 
posting of images and video. Given the findings of this study, the decision not to allow participation is of 
interest as it suggests there may be systemic responses to the use of social media, even with considerable 
research safeguards in place that respected the anonymity, confidentiality, and ethics. The response of the 
school boards potentially also reflects a generally negative view of the use of social media by educators. 

Despite the lack of consent from the district school boards, school leaders from two other districts heard 
of the study and participated anonymously. Within the NING, all of the participants used avatars and were 
instructed on protecting the identity of schools, districts, students, and themselves through the careful 
reporting of cyber events and responses. 

3.3 Data Collection and Analysis 

All of the data from the Limesurvey were compiled and are reported next in the Findings. For the responses 
to the questions in the NING, two levels of analysis were applied. First, the responses to each question in the 
NING were compiled. Next, the compiled data was analyzed to note reflective passages, key quotations, and 
patterns and themes in the vice-principals responses (Cresswell, 2012). At that level of analysis key themes 
and codes were identified, then organized to make sense of the data. For purposes of this paper, some of the 
key questions from the NING are reported in the Findings section which follows. At the final level of 
analysis, the findings were reviewed relative to the relevant literature, leading to recommendations presented 
in the discussion section of this paper. 


4. FINDINGS 

This study used a mixed-methods approach to data collection with an emphasis on qualitative data collection. 
Ten respondents completed the Limesurvey questions; however it was believed that one participant 
completed the survey twice. For the purposes of this study, it was decided to include the data rather than 
speculating about the respondent’s double entry. Three male and 7 females responded. Six participants were 
between the ages of 35 and 45 and 4 were between the ages of 45 and 60. Seven participants had between 
zero and three years in their role. Three vice-principals had between four and six years of experience in their 
role. Most were fairly new at the role of vice-principal. The participants were also asked to indicate their 
degree of knowledge with the recently legislated Bill 13, the Accepting Schools Act (LAO, 2012) which 
requires schools to respond to any harmful cyber events affecting students and school community which are 
raised to their attention. Knowledge of the legislation was diverse as five respondents had limited knowledge, 
two respondents had some knowledge and three respondents had considerable knowledge. 
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4.1 NING Data 

While the online survey was completed by all participants, the number of participants reading and posting 
online in discussions varied from week to week. For the most part, participation reflected the chronology of 
the postings; however, some respondents went back to prior postings from previous weeks or contributed to 
depth probing questions at different times. 

NING Question: “How has Bill 13 affected the work you do in schools? What other policy or legislation 
has affected your work? In what ways? What policies or rules do you think about when you are resolving 
safe school issues? ” 

The respondents positively viewed the Accepting Schools legislation in two ways. To some, the 
legislation provided helpful and explicit information that could be shared with students and parents. The 
school leaders used terminology such as “clear direction” and “guidelines.” One vice-principal said that “Bill 
13 has given school administrators clearer direction and guidance when dealing with students.” Another 
vice-principal said that the policy provided support for what was considered good practice, explaining in this 
way: 

[The policy] defines bullying explicitly and it is helpful to have a detailed definition to which I can 

refer. This legislation compels the administration to notify both the parents of the student harmed as 

well as the student who has harmed someone ...something that is ingrained in our practice. 

For the most part, the vice-principals did not report that other policies were considered in the context of 
negative online behaviour and cyberbullying. Bill 13 was widely perceived to be a helpful policy both for 
the reinforcement of current practices as well as for support with future decisions. One participant said the 
policy provided guidance for “creating a safe school environment and responding to the activities of 
students despite location and time of day.” 

NING Question: “What in your view is cyberbullying? What does this term mean to you in relation to 
your experience as a VP? How is it distinguished from other negative online communication? ” 

One school administrator reported that, “negativity and bullying has gone on since Socrates taught, but 
cyberbullying adds a new dimension to the role of the vp.” Throughout their postings, the vice-principals 
made distinctions between cyberbullying and online conflict. The respondents provided very clear terms of 
what they believed constituted cyberbullying citing the repetition of events , power imbalances, and 
harassment as its characteristics (similar to the description in the policy). Cyberbullying, as one respondent 
explained, “uses online means and applications to harass another person. This can be in the form of direct 
communication with the victim (emails, chats, posts on their apps) or online communication online about that 
person.” Another administrator indicated that “Cyberbullying is the use of technology to repeatedly 
intimidate and harass someone.” 

The vice-principals in the study indicated that cyberbullying included direct and indirect references to 
another student whether that student was named or not. While intent was identified as an aspect of 
cyberbullying, it was also mentioned that a lack of intent did not preclude it. Vice-principals indicated that 
there were both positive and negative aspects to social media and its use by students. This had complex 
implications for their role as administrators because they said that they spent “an inordinate amount of time 
‘sorting and sifting’ information on social media.” The administrators noted in their comments that social 
media escalates the harassment by its anonymity and brings shame to the student who receives it. There is a 
“profound impact if it is not discovered.” The potential for public shaming was cited as a great concern as 
“young people are given so much power in social media to broadcast their unkind thoughts that might have 
been either whispered or scribbled on a passed note in the past.” However, while acknowledging these 
negative aspects to online communication, the school administrators ironically noted that online 
communication nevertheless leaves “hard evidence” or a “digital footprint” that they could follow and use in 
their investigations. In summary, vice-principals could readily distinguish cyberbullying from other kinds of 
negative online communications, and were able to follow the evidence because online misconduct leaves a 
digital paper trail. 

NING Question: “How do you perceive your role as an administrator in the context of Safe School 
legislation like Bill 13?” 

When they reflected on their roles as administrators in the context of the legislation, the vice-principals 
repeatedly reported feeling a sense of duty or “obligation.” They took their responsibilities seriously in the 
context of the legislation as well as how they saw their roles based on the expectations of parents. They also 
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saw the negative and positive aspects of social media. They spoke of the obligation they felt to create a 
welcoming climate in their schools and to keep kids safe. One respondent stated, 

Our role as Administrators is always one of providing all students with the opportunity to come to 
school and feel safe and successful. With respect to cyberbullying they reported that their role 
included prevention as well as investigation of cyberbullying, in order to care for kids and help them 
learn. 

Another articulated that it was reasonable for parents to expect that when students were in school they 
would be safe. One administrator felt their role “is to continue to support students through the social media 
landscape, help them understand when they fall outside the lines of what is deemed appropriate, and hope for 
learning and better choices for the next time.” This was echoed by another vice-principal who spoke of the 
potential of social media and other technology to educate. While this respondent spoke of hearing about the 
negative actions of students on social media, the observation was made that, “however, other students use 
social media responsibly.” The vice-principal elaborated stating that a “finite group of students that 
‘overshare’... lack understanding of their actions.” Again, this was an indication that some vice-principals 
perceive that many students do use and understand social media more carefully. In the context of Bill 13, 
vice-principals see their role as a duty or obligation to keep their school communities safe and to help 
students learn to communicate appropriately. 

NING Question: " Reflect upon the nature of an online community and its relationship to your school. 
You may consider your own experience with online communities (this one or another) as well. ” 

The school leaders spoke of the negative and positive aspects of online communities relating to their work 
and their personal use. One respondent felt that online communities were helpful in providing an opportunity 
to “learn from other educators through online discussion, blogs, and connecting with others in similar roles to 
mine.” The vice-principal also saw this kind of communication as an alternative to what was perceived to be 
a decline in the funding of professional development. However, this respondent did not use the NING for 
these purposes and posted infrequently. One vice-principal acknowledged the potential for the positive use of 
social media as a communication link between schools stating, “This poses a timely question. Honestly, I 
have avoided being part of any social media network that offers the opportunity ‘to see’ my personal life.” 
Another respondent acknowledged the positive aspects of online communities such as Twitter, Vine, and 
Instagram for school communication. However, the darker side of social media, including the surreptitious 
filming of staff and administration, was also commented on, as was the degree of online humiliation that the 
victim receives, whether that victim is a school staff member or, of even greater concern, a vulnerable 
student. This participant stated. 

Students are connected through their own online communities which are often the new “Wild West” 
where stray online bullets regularly wound its inhabitants through online insults, ‘beefs’, etc. As 
administrators we are asked to try to enforce law and order in these online environments. 

In summary, when they reflected on the nature of online communities and their relationship to schools, 
vice-principals perceived and identified positive and negative aspects to these kinds of communication but 
they expressed hesitation to participate in them. 


5. DISCUSSION 

The review of the literature on social presence indicates that electronic media have been changing 
relationships for four decades (Korzenny, 1978; Short et al., 1976; Walther, 1992) leading to investigations 
of social presence as a quality that is impacted by the lack of proximity in place and time but can be 
addressed in online courses (Garrison et al., 2000). While researchers argue that social media interaction can 
evoke at least the vestiges of community (Kobler et al. 2010) and interaction is both immediate and intimate 
(Rettie, 2003), the feeling of being in a community is impacted by cyber-misbehaviour. The vice-principals 
in the study saw is the positive and negative aspects of the social media in their investigations and did not 
pursue its personal use. Nor did they use the NING as a means of free expression. 

Online communication changes relationships and communications in schools, impacting school climate 
and school safety. The participants in the study spoke of the challenges associated with the anonymous 
quality of cyberbullying and harassment, a phenomenon thought to be the underlying cause of cyber 
aggression (Mark & Ratliffe, 2011). Power imbalances appear to be connected to social roles and a lack of 
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social regulated presence (Yeh, Lin and Lu, 201 1). The school leaders in this study see that it is of paramount 
importance to discover incidents of cyberbullying because of its “profound impact” on students. However, 
they also asserted that an “inordinate” amount of their time was spent investigating what some 
vice-principals felt was relational conflict and not cyberbullying. Of interest to this study was the breadth of 
restorative practices outlined, including conversations, questions, and impromptu conferences that were used 
in an attempt to offset anonymity and aggression. 

The findings in this study indicate that vice-principals see that they have an obligation to keep schools 
safe and to create an inclusive and welcoming climate. There was evidence in the study that this 
responsibility felt by the vice-principals to maintain safe schools predates the existence of cyberbullying 
policy (Bill 13) and vice-principals extend its scope to include online conflict as well as a legislated response 
to cyberbullying. In holding this responsibility, however, the vice-principals do speak of a “new dimension” 
to their role and the amount of time spent navigating through social media. Vice-principals work to rebuild 
elements of community and social presence when they investigate and respond to incidents of online 
misbehaviour. The more positive, regulated aspects of community, achieved through restorative practices, 
help to respond to a lack of social and regulatory presence reflected in social media communications. 
Participating vice principals did not articulate the term social presence per se, but they did suggest that the 
anonymous quality of online communication could create the conditions that exacerbated peer-to-peer 
conflict and bullying. 

Vice-principals connect social media to their roles as school administrators, likening their online social 
presence to the vigilance of the sheriffs in the Wild West, who by their presence “enforce law and order in 
these online communities.” They are reticent to use social media for other purposes, such as for personal use 
or had stopped using it entirely. Some vice-principals saw the value of Web 2.0 technologies for the purposes 
of professional development but did not view the NING in this study in this way. As they immerse 
themselves in technologies for their roles, they maintain a distance and wariness when it comes to their 
personal use of technologies. In the words of one vice-principal, their school experiences “jaded” them. It is 
hoped that these findings can be followed up and confirmed or explained further in future studies. 


6. CONCLUSIONS AND RECOMMENDATIONS 

While small in scope, this study may inform or guide future studies. The vice-principals in this study reported 
that they enact policy which supports safe schools, and they see their role as maintaining school communities 
as safe places. While they engaged with and used Web 2.0 technologies to respond to online aggression and 
cyberbullying, events that had a “nexus” in their schools, they could see the negative and positive aspects of 
these media. One aspect of their role which may not yet be recognized is that secondary vice-principals also 
are teaching about appropriate online behaviours. They saw students’ missteps in the online world (such as 
oversharing) as opportunities or teachable moments. The vice-principals used social media for investigations, 
another area that could be considered in future study as it was not present in the literature reviewed. While 
they use these technologies in the day-to-day context of their jobs and for the investigation of online 
incidents, they would appear to use them with great reluctance for their personal use and this may extend to 
professional use. 

There was a time in the not-too-distant past where school administrators focused their discipline-related 
responses to events and misbehaviour that happened during school time or on school property. This study 
indicates that the vice-principals in this study are responding to cyber events which occur off school property 
and outside of class hours in order to maintain student safety. This may help to explain the hesitation of some 
of the vice-principals in this study to use these same technologies beyond the scope of their school day. The 
Digital Wild West remains as space worthy of interest and further study. 
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ABSTRACT 

This research described here examines the role of e-health and medical informatics through the lens of one e-health 
knowledge broker in Canada. Eating disorders are an important issue in Canadian health and it is difficult to find 
accurate information online. Theoretical models examined include those which describe the roles of health knowledge 
brokers, including those who advocate for critical social change. A case study is presented of one Canadian agency which 
brokers knowledge about eating disorders. This agency has a highly complex role which employs multiple technologies: 
telephone help lines, a website, and webinars. The help lines and email supports are designed to respond to questions 
about eating disorders and to refer individuals to service providers. When compared to the oft-cited role of the knowledge 
broker to disseminate research information, the key roles of this agency move beyond this, providing reliable and 
consistent information; brokering spaces for capacity-building; promoting crtical health literacy and advocating for health 
equity. 
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1. INTRODUCTION 

The role of the e-health knowledge broker agency has not been fully explored in the literature, leaving gaps 
with respect to both theoretical considerations and rigorous research studies. This paper presents a case study 
of one health knowledge brokering institution which maintains both e-health and traditional knowledge 
brokering practices. A case study is provided which introduces the complexity involved in measuring the 
impact of health knowledge exchanges in general and e-health knowledge brokering in particular which is 
designed to support the prevention, intervention, and treatment of eating disorders in Canada. This research is 
presently in its initial phases. 

First, some key questions help increase the focus on the issues. In the present digital era, where do 
individuals turn to find information when they suspect that their own health or the health of someone who is 
close to them is in jeopardy? How does this change if there is a culture of silence or shame attached to this 
ailment? What are society’s imperatives if this particular health condition can be fatal? In these 
circumstances, what are the key roles of the health knowledge broker and how can we measure the impact of 
the knowledge broker? 

In the case of the provision of health information for eating disorders, the role of the e-health knowledge 
broker takes on a significance and immediacy that may not be experienced by other knowledge brokers or 
knowledge mobilization intermediaries. The role of the knowledge broker in this case is imbued with a sense 
of urgency to provide current, helpful, and factual information to support the efforts of the individuals 
seeking help to connect with health care providers, but also carries a responsibility to educate society. 

According to their website, www.nedic.ca. The National Eating Disorder Information Centre (NEDIC) is 
a Canadian non-profit organization providing resources on eating disorders and weight preoccupation. 
NEDIC defines clinical eating disorders as medical conditions which require treatment from health 
professionals and these include anorexia nervosa, bulimia nervosa, binge eating disorder, and eating disorders 
not otherwise specified. Traffic to the NEDIC website is reported to be very high, and in addition to this, 
approximately 1,600 individuals annually contact the NEDIC helpline by phone or through email. This paper 
begins to explore the work of NEDIC in the context of research on the role of knowledge brokers in general, 
and e-health knowledge brokers in particular, recognizing that they provide a vital health education space. 
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1.1 Eating Disorders - the Canadian Context 

Within the past year, one of the reports of a Canadian parliamentary committee has raised serious, multiple 
concerns with respect to the rising incidence and lack of treatment for Canadians with eating disorders, 
especially girls and women (House of Commons, 2014). According to the expert witnesses, eating disorders 
are described as a serious form of mental illness impacting the health of 600,000 to 990,000 Canadians at any 
point in time. One expert witness testified that anorexia nervosa and bulimia nervosa are fatal for 1,000 to 
1,500 Canadians annually and this rate of incidence may possibly be underreported (House of Commons, 
2014). 

Eating disorders are also an equity issue; anorexia has the highest rate of mortality of any mental illness 
but treatment wait times in Canada for eating disorders are not responded to with the same urgency as other 
life-threatening health crises (House of Commons, 2014). Four out of five persons affected by eating 
disorders are women, which may reflect that society pressures girls and women in numerous ways to obtain a 
certain ideal body type. Perceptions of body size and shape are impacted by society’s stereotypes, including 
the marginalization and stigma surrounding weight which can prompt disordered eating. Expert Canadian 
witnesses identified that fat stigma, which encourages food regulation and dieting, and a national culture of 
blame for individuals with eating disorders and their families are barriers to treatment (House of Commons, 
2014). 

According to Jasper (2005) if the generally-accepted and relatively unchallenged messages of a society 
state that the ideal body is achievable with effort, when in fact this body type is generally unattainable for the 
average population, this contributes to weight preoccupation. Schools are particularly problematic 
environments for messages about weight and appearance because peers can reproduce societal stigma against 
obesity, and they can also encourage individuals to pursue diets and persist with them. There is, in addition, 
an overall focus on healthism in schools (Rich, Holroyd and Evans, 2004) which focuses on a person’s 
individual responsibility for their health and size. If this discourse around personal responsibility and health 
is not interrogated and deconstructed for students, it can help to build anxieties about body size and shape 
(Rich etal., 2004). 

There is, in addition, societal pressure and stigma associated with fat. Schools have been identified as 
“potentially toxic environments” for weight and appearance messages (House of Commons, 2014; Robertson 
1 & Thomson, 2014). Peer messages can reproduce stigma against fat, but they can also encourage those 
who are dieting to continue to lose weight. If students are perceived to be overweight, they are more likely to 
experience bullying and teasing (Fox & Farrow, 2009; Puhl, Luedicke & Heuer, 2011) which can, for 
example, impact their participation in physical education programs (Puhl & Luedicke, 2012). Research has 
also shown that teachers and in particular, physical education teachers, hold stereotypes around weight; 
holding lower expectations for students who are overweight in both their cognitive and physical areas of 
potential (Greenleaf, Martin & Rhea, 2008; O’Brien, Hunter & Banks, 2007). 

These kinds of oppressions occur more easily in a society which focuses on individual responsibility and 
lifestyle change to look a certain way rather than looking at how society is influencing or determining health. 
Canadian health policy has been criticized for drawing too heavily on lifestyle change or a neo-liberal focus 
rather than examining societal determinants of health (Mikkonen & Raphael, 2010) and also for pursuing a 
medical anti-obesity model which promotes consumerism through diet foods, regimes, and self-discipline 
(Rail & Beausoleil, 2003). 

In summary, then, eating disorders in the Canadian context present a very real health risk, but the 
prevention, intervention and treatment of eating disorders has been compromised by health care systems and 
education systems that promote efforts to lose weight, and equate health with certain sizes. The key 
implication for a health knowledge broker in this particular context is that their messages are, essentially 
counter-cultural. The health knowledge broker needs to convince both the medical practitioner community 
and society that food and weight preoccupation are harmful and potentially fatal to certain populations. 
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2. THEORETICAL CONSIDERATIONS 
2.1 Knowledge Brokers 

The transfer of health research to practice has been described as a slow and complex process (Graham et al., 
2006). Knowledge brokers are people or organizations who help to package the knowledge and create 
connections between the researchers and their audiences (Meyer, 2010). Knowledge brokering is growing in 
the postmodern era and because new knowledge and brokering spaces are growing, in particular in the 
healthcare sector, this health research can have immediate implications for patient care and treatment. While 
earlier descriptions of the role of knowledge brokers include the facilitation of the “creation, sharing and use 
of knowledge” (e.g., Svarrisson, 2001), Meyer provides other examples of knowledge brokering in healthcare 
which delineate additional roles for knowledge brokers: 

Identifying knowledge 

Redistributing and disseminating knowledge 

Rescaling knowledge 

Transforming knowledge 

According to Meyer, in order for health knowledge to be used, it is not just moved along but it is 
re-packaged to meet the needs of the users. In order to do this, Meyer theorizes that knowledge brokers need 
to live and work at the peripheries of both the research world and the world of the knowledge user, working 
behind the scenes in collective and interactive ways. The end product is “brokered knowledge” (p. 123) 
which is knowledge “made more robust, more accountable, more usable; knowledge that ‘serves locally’ at a 
given time; knowledge that has been de- and reassembled” (Meyer, 2010, p. 123). 

The Canadian Health Services Research Foundation (CHSRF) (2003) see knowledge brokers as 
go-betweens and they emphasize the human aspects of that role. Brokering means bringing people together 
who share common issues in order to share their experiences and to hear about new evidence. In their words, 
“Getting the right mix of people and information together to tackle the right issue at the right time is the 
essence of brokering” (p. 9). Brokering also involves building networks and relationships both to share 
current research and to stimulate future research (CHSRF, 2003). 

CHSRF concludes that health knowledge brokering is not a commonly found role in Canada. Good 
brokers are curious, flexible, well-informed and able to handle a range of information and ideas. They are 
also the ones who will resist quick fixes until they have seen that the research-based evidence has been 
reviewed. CHSRF advocates that the tasks of the broker include the following: 

bringing people together to exchange information and work together; 
helping groups communicate and understand each other’s needs and abilities; 
pushing for the use of research in planning and delivering healthcare; 
monitoring and evaluating practices, to identify successes or needed changes; 
transforming management issues into research questions; 
synthesizing and summarizing research and decisions (CHSRF, 2003, p. i) 

While the CHSRF report sees multiple roles for knowledge brokers, the report also acknowledges gaps in 
the research. For example, networking is a key function but much more research is needed on how 
knowledge brokers set up networks, provide information and train individuals within their organization. 
Ward. House and Hamer (2009) concur, stating that more needs to be investigated as one challenge to 
knowledge brokering is a lack of understanding of how it works based on different contexts. 

Early research and concept development on the role of the health knowledge broker indicates that more 
specific research is needed to understand the contexts within which knowledge brokers operate. Early 
theoretical frameworks focused on knowledge producers and users as two camps, linked by the knowledge 
brokers. More recent models indicate that knowledge brokers transform the knowledge to specific contexts, 
and that they work with a presence in both the world of the researcher and the world of the knowledge user. 
In the section which follows, e-health knowledge brokering is explored, opening up new aspects of the role 
of the e-health knowledge broker. 
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2.2 E-Health Knowledge Brokers 

The term and concept of e-health has been defined by Eysenbach (2001) as the provision of health 
information and health services enhanced by utilizing the internet and other technologies. He theorizes that 
access to multiple sources of information and research can not only inform but empower patients, give them 
more choices, and encourage more patient-centered medicine. It can also extend health care to persons who 
may be geographically distant, or who seek information during off hours. E-health provisions are especially 
important for persons who perceive a stigma associated with their illness. An extensive US study of internet 
use for e-health concerns indicates that the internet information can be a valuable tool for education about 
health, particulary when the health concern is a stigmatized illness (Berger, Wagner & Baker, 2005). 

Connecting health care practitioners to persons who are potentially at risk is a key role in e-health 
provision today. An e-health information broker can be a repository for practitioners and provide access to 
current research. An e-health information broker can also create an interface that encourages people seeking 
help to connect with health care practitioners. Health care practitioners also have a role in helping their 
patients identify reliable e-health information websites, or teach them how to critically assess the health 
information which is on the web. 

Adolescents, in particular, can be at risk for health concerns where there is stigma attached (such as 
mental health) because of their reluctance to reveal their concerns and seek out a health care professional. 
One Canadian study of how adolescents use the internet for health information reveals that topics of body 
image and nutrition are very important to adolescents and these are some of the most frequently accessed 
spaces on the internet. Participants in the Canadian study indicated that adolescents frequently access 
websites for health information and they value the anonymity provided by a webspace to share their health 
concerns (Skinner, Biscope, Poland and Goldberg, 2003 ). 

Early research on the internet and its impact on the provision of health care knowledge indicated that even 
at the turn of the century, literally millions of biomedical research articles were put online annually, creating 
concerns for information overload and questions surrounding how knowledge users can determine the quality 
of online information, prompting a call for more formal evaluations of e-health websites (Christensen & 
Griffiths, 2000). More recently, studies have focused on examining how health users make use of the 
website information, resulting in new understandings about the factors which contribute to trust in health 
websites (Sillence, Briggs, Harris & Fish wick, 2007). 

One study which evaluated web-based information on eating disorders created a scale of evidence-based 
clinical practice statements and then analyzed websites based on this scale. They found that in general the 
quality of information available online about eating disorders left room for improvement, and more research 
was needed to determine the types of information and quality indicators that health users would find to be the 
most helpful (Murphy, Frost, Webster & Schmidt, 2003). 

2.3 Measuring Health Knowledge Outputs 

Very little has been written about how to measure the impact of a health knowledge information on the web 
(Murphy et al., 2003) and in addition, little is known about how to measure the impact of a health knowledge 
broker. One output tool which as designed to measure health research (not broker) impact has been proposed 
by Kuruvilla, Mays, Pleasant and Walt (2006). Its four dimensions are areas of research impact: research, 
policy, service and society. 

• Research - related impacts include publications and papers; 

• Policy impacts include, for example, policy networks, and participation at various policy levels; 

• Service impacts include quality of care, and evidence -based practice; and 

• Societal impacts include, for example, health literacy, equity and human rights, social capital and 
empowerment 

Health literacy in this context is described as as a broad range of skills which can help to locate, 
understand, assess and utilize health information for informed choice, reduced risk, and increased quality of 
life (Kuruvilla et al., 2006). This impact framework has not yet been applied to outputs of health knowledge 
brokers or e-health knowledge brokers but it could be potentially helpful in examining and reporting the 
effectiveness of e-health brokers. 
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Nutbeam (2000) also argues for health literacy, finding that health education programs which focus on 
telling people what to do or not to do have not been successful in producing health behaviour change, and 
transmission has not been an effective way to moderate health outcome gaps between different groups. He 
describes health literacy as a set of skills that lead to enablement. Basic or functional health literacy 
encompasses knowledge and skills; communicative health literacy includes more advanced skills which can 
be applied in new situations and communicated. The third level, critical literacy , involves the critical 
analysis of information which empowers individuals to take more control over their health and to influence 
society (Nutbeam, 2000). As Kuruvilla et al. (2006) note, human rights and equity include the rights of 
individuals to shape health decisions that impact their lives. 

In summary, theorizing on the role of the knowledge broker in health knowledge translation indicates that 
the role is both complex and critically important. At stake is the empowerment and enablement of the users 
of health research knowledge, the building of community and capacity surrounding health knowledge 
exchanges, and improvement of the health and life chances of individuals. 


3. RESEARCH DESIGN 

This paper describes proposed research using the lens of a descriptive case study. Lichtman (2012) describes 
the case study as a research approach which involves a specific and detailed study of an entity. Qualitative 
case study research is described by Merriam (2009) as a distinctive form of research design within the critical 
epistemological perspective which assumes that there are multiple realities, dependent on different 
individuals’ world views and experiences, which involve the privileging of certain realities. For example, 
within a case study approach in the critical paradigm, there is an examination of those whose interests are 
protected and preserved at the expense of other perspectives (Merriam, 2009). Similarly, Guba and Lincoln 
( 1994) explain that the reality in a case study which is described to the researcher(s) is shaped by forces such 
as “social, political, cultural, economic, ethnic, and gender values” (p. 109). 

By design, the methodology proposed here is distinctively dialectical and hermeneutical, seeking 
information and refining it through the interactions which occur between and among participants and 
investigator and become refined as a result. The voice of the researcher is considered to be both advocate 
and activist, “transformative and intellectual” (Guba & Lincoln, 1994, p. 112). 

Data collection for the case study presented here will employ multiple sources such as online offerings, 
webinars, interviews, observations, e-documents, and artifacts. These multiple sources allowed for deeper 
understandings and confirmation of findings. Data will be analyzed through the identification of key themes 
across the compiled data (Cresswell, 2012). Rather than employ the structure of the entry and closing 
vignette (Cresswell, 2012, p. 106), this planned research is introduced through a case study which outlines 
the context, the issues and theoretical conceptions of the role of the e-health knowledge broker. A discussion 
of selected issues and assertions follows. 

3.1 NEDIC: A Descriptive Case Study 

NEDIC identifies itself online as, “Canada’s leading source of information and support in the field of eating 
disorders and related issues” (www.nedic.ca). It is a not-for-profit organization that has been working since 
1985 to promote health and raise awareness of the complexities surrounding food and weight preoccupation. 
They provide a first point of contact. Of the individuals who contact NEDIC over the course of a year, 
approximately 60% will be individuals who have an eating disorder but their parents, friends and teachers 
will also call for information. 

Education, outreach and client support are three of the central roles that NEDIC undertake as an e-health 
knowledge broker. They maintain a presence through email, phone lines and their website, as well as through 
webinars and events. NEDIC provides information to school counsellors, teachers, psychologists, 
psychotherapists, family doctors and researchers, among others, but central to their mission is to give timely 
information to individuals who need more information about eating disorders and their families. It is 
important that individuals who contact NEDIC for support are met with consistency, with hope and with an 
expectation that NEDIC can empower them to navigate to seek answers. 
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NEDIC connects individuals with information on service providers in their vicinity; they maintain the 
Service Provider database with just under 800 program providers, including those who are provincially 
funded, fee-for-service or free. Because the number of institutional beds for individuals with severe eating 
orders is inadequate in Canada (House of Commons, 2014), many individuals between early identification 
and tertiary care are not having their needs met. In some of these cases, because individuals have multiple 
concerns when they have an eating disorder, NEDIC may be able to help them to connect with service 
providers in order to meet another of their health needs. 

A key issue for maintaining the centre is funding - they have limited resources and a huge mandate. 
Approximately 20% of their funding comes from the Ministry of Health. Other sources of income include 
fund-raisers, the biennial conference and grants from foundations and sponsors who are carefully selected 
based on hospital criteria and philosophy. 

NEDIC maintains a database of information on eating disorders based on references to peer-reviewed 
material. While their funding permits access to eating disorder journals, they do not have the funds or 
personnel to scan journals in other areas such as clinical psychology or school counselling. To broaden their 
understandings, NEDIC gathers information through listservs, seminars that they attend and many informal 
conversations with clinicians, teachers and other activists in the field. 

NEDIC offers a biennial conference which takes months of planning. Significant time is given to the 
identification of key researchers and cutting edge research in eating disorders and related areas. NEDIC also 
provides information through an internal listserv and gives and receives information through the individuals 
who present webinars for them. Each of these webinars sold out within two days of opening the enrolment. 
They organize communication efforts by considering different populations - universal populations for 
prevention, an early intervention group and those who identify as having eating disorders who need to find 
help. 

Two projects undertaken by NEDIC for universal populations have been directed at schools through the 
Beyond Images curriculum, and at Girl Guides through the Love Yourself badge. Beyond Images is a 
body-image and self-esteem curriculum designed for students in Grades 4-8 - a time of vulnerability for 
student populations because their bodies are changing. This curriculum was designed in 2009 and since 2013 
is available free of charge through the NEDIC website. NEDIC estimates that, conservatively, more than 
10,000 students have undertaken this curriculum. Similarly, the Love Yourself badge, which NEDIC 
developed with Girl Guides, is the 2 nd most popular challenge badge in Guiding with approximately 1,000 
guide members undertaking it annually. 

At the present time, a key focus for NEDIC as an e-health care provider is to undertake rigorous research 
to document their role and their impact as an e-health knowledge provider. To that end, an investigation has 
begun to interview key personnel who access NEDIC as an e-health knowledge broker to better understand 
the roles that NEDIC provides and examine how researchers, practitioners and the public use the e-spaces 
and physical spaces for knowledge brokering that NEDIC creates. 


4. DISCUSSION 

Health knowledge brokering functions can include linkages and partnerships, awareness, capacity-building, 
support, and policy influence (Cooper, 2012). While NEDIC performs many of the recognized roles of the 
knowledge broker; this organization also places an emphasis on e-health brokering in non-standard areas. For 
example, they appear to be more client-centered than research-centered, but maintain deliberate standards for 
dissemination of research knowledge. They also maintain a focus on collective knowledge without 
privileging one type of information at the expense of another. Dede (1999) has identified that collective 
learning is facilitated when knowledge mobilization efforts are supported by digital technologies. NEDIC 
uses digital technologies through email, webinars and websites for the stated goal of capacity building, but 
are seeking ways to measure the outcomes of their health knowledge brokering. 

This review of the literature and presentation of the initial case study indicate that there is much to be 
gained from research which examines the role of e-health knowledge brokers in general, but, in particular, in 
examining the role of an e-health knowledge provider who maintains a deliberate equity stance both by its 
funding imperatives, but also prompted by the specific needs of Canadians with eating disorders who are 
experiencing inquiries in health care and health outcomes. Much more needs to be understood about the role 
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of an e-health provider which expresses a commitment to building critical health literacy understandings. 
While hope and empowerment are not often found in the literature as knowledge brokering outcomes, they 
are indications that findings related to this particular e-health knowledge broker will influence the present 
field in new ways. 
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ABSTRACT 

The purpose of this study was to examine learning preference — the match between learners and learning methods — and 
students’ information behaviour in technology-rich information environments. The major question asked was: Elow will 
high school students’ information behaviour differ by gender and academic interests? A total of 88 students (37 girls, 51 
boys) from a predominantly African American high school district in the south of the United States participated in the 
study. The Integrated Communications Technology Learning (ICTL) survey was used to examine differences in high 
school students’ learning preference for information seeking, information sharing and classroom learning. High school 
girls and students with Science, Technology, Engineering and Math (STEM) interest were found to be significantly more 
positive toward learning in the traditional classroom. 


KEYWORDS 

Information behavior; Learning preference; High school 


1. INTRODUCTION 

Weiler (2005, pp. 46) posited that “the entire world, is currently in the middle of a massive and wide ranging 
shift in the way knowledge is disseminated and learned.” Information and Communication Technology (ICT) 
tools now offer previously unavailable options for interaction with information for informal-to-formal 
learning. Additionally, many students use ICT options for rapid information access, messaging, and online 
connection in their personal lives and many would welcome opportunities to use these networking tools for 
formal-to-informal learning (Rennie & Morrison, 2013). Yet, even as there is a trend towards online learning, 
there is little research on the effect of match or mismatch of learners to their preference of learning methods 
for learning in traditional and technology-mediated learning environments (Curry 1981). The rapid growth of 
personal technologies throughout the world and the rapid emergence of mobile technologies and applications 
(Henrlquez & Organista, 2012; Khaddage & Knezek 2012) increases the need for studies that can help clarify 
how students view the use of technology-mediated communications tools for knowledge construction and 
how students’ views may relate to learning outcomes. Research on the psychology of learning indicates that 
learner attitudes relate to learning outcomes (Driscoll. 2005). Additionally, learning preference has been 
linked to academic performance (Koch et al., 2011) and to motivation for learning-related activities (Hong & 
Milgram, 2000). This study of the learning preference of students in one school district is part of the authors’ 
ongoing research on the role of student learning preference to academic goals. This study also includes an 
examination of the reliability and consistency of the Integrated Communications Technology Learning 
(ICTL) survey to gauge high school students’ learning preferences for three categories of informal-to-formal 
learning interaction — classroom learning, ICT information seeking, and ICT information sharing. 
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2. CONCEPTUAL RATIONALE 

2.1 Learning Options 

Communication is an integral aspect of learning (Warren & Wakefield. 2012). ICT tools offer previously 
unavailable options for interaction with information for informal-to-formal learning. As the newest personal 
technology tools blur the line between informal and formal learning (Cox, 2012; Mills, Knezek & Khaddage, 
2014; Henrfquez & Organista, 2012; Khaddage & Knezek, 2012), mobile technologies are gradually 
becoming more geographically dispersed (Castells, 2011) and more readily available to students. The many 
new ICT tools provide a choice in options for learning. Student choices and preferences are related to 
learning and they warrant examination of how students prefer to interact with information and the match 
between learning options and learning methods (Keefe, 1979; Curry, 1981; Owens & Straton, 1980). Interest 
in ICT tools for learning is shifting from a focus on what channels and resources are used to an emphasis on 
how students will encounter, interact, and interpret information that can be sought out with ICT tools (Ilgaz, 
Mazman, & Altun, 2015). 

2.2 Learning Preference 

Orhun (2013, p. 1159) stated, “Learning can be expressed as gathering information, processing information, 
the improvement of thinking, and the method of selection for attaining knowledge.” Student preference for a 
mode of learning is an important variable in the effectiveness of learning (Owens & Straton, 1980). Learning 
preference is a facet of how we learn. It has been defined in relation to learning style and methods, as how a 
learner perceives, interacts, and responds to learning opportunities (Keefe, 1979). Mayer and Massa (2003) 
examined learning preference along with aspects of cognitive ability and cognitive style. They describe 
learning preference as a distinct aptitude and “property of the learner’s interaction with a particular learning 
situation” (Mayer & Massa, 2003, p. 838). While studies on learning styles have not provided conclusive 
evidence regarding the effect of learning style on academic performance, Orhun’s (2013) research revealed 
that preferred learning style is potentially a tool for improvement of mathematic performance. Learning 
preference relates most directly to how students prefer to attain knowledge and was found to be a predictor of 
academic performance in a study of nursing students, for students those who reported a preference for 
multiple approaches for learning (Koch et al., 2011). Additionally, Hong and Milgram (2000) reported that 
learning preference is related to motivation for homework and out-of-school learning. 

2.3 Research Questions 

The research questions addressed were related to how high school students like to learn. Students’ 
information behavior (information seeking and information sharing) in technology-mediated ICT learning 
environments, as well as students’ liking of classroom learning were examined by gender and academic 
interest in Science, Technology, Engineering and Math (STEM). 

The specific research questions examined were: 

1 ) How are male and female students different in their learning behaviors with regard to information? 

2) How are students interested in seeking STEM and non-STEM courses of study different in their 
learning behavior with regard to the information? 


3. RESEARCH METHODS 
3.1 Participants 

Institutional Review Board approval was granted for this study of learning preference among high school 
students. Data was gathered using paper and pencil surveys. All high school students, with the exception of 
those who were not eligible for a school field trip, from the district’s only high school were invited to 
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participate in the study. Eighty-eight of the 296 students who attended the field trip returned permission 
forms and surveys and were therefore included in the study. Participants were 58% male and 37% female, 
attending high school grades 9 to 12. Fifty percent of students, n= 44, indicated that they are interested in 
seeking a STEM-related career. 

3.2 Measurement and Instruments 

The Information and Communications Technology Learning (ICTL) survey was designed and validated to 
address questions relating to students’ preferences in utilizing ICT and to assist in understanding individual 
differences in information behavior. Instrument development included analysis for internal consistency 
reliability, principal components exploratory factor analysis, multidimensional scaling, and higher order 
factor analysis (Mills, Knezek & Wakefield, 2013). The original version of the ICTL survey consisted of 16 
items to assess three constructs: two of the three are information seeking and information sharing (Author, 
2014). A new construct, classroom learning, emerged in analysis of the ICTL for the participants of this 
study. These constructs were measured on a five -point Likert scale ranging from (1) “strongly disagree” to 
(5) “strongly agree.” Exploratory factor analysis indicated an original model for constructs wherein the 
information seeking construct was developed from six items (ic8, iclO, icl 1, icl4, ic2, ic 1 3), the information 
sharing construct was created from six items (ic3, ic6, icl5, ic4, icl, ic 1 6), and the classroom learning 
construct was created from four items (ic9, icl2, ic5, ic7). 

Confirmatory factor analysis was performed to validate the scales from exploratory factor analysis. To 
modify the model, the observed variables that had produced low factor loading (<0.5 ) were eliminated. The 
modification indices were consulted for the model modification. Figure 1 presents the final measuring items 
for each construct and the factor loadings. The models were considered acceptable if CFI > 0.90, GFI > 0.90, 
AGFI > 0.90, RMSEA < 0.08, and SRMR < 0.10 (Joreskog & Sorbom, 1993; Hu & Bender, 1999). Table 1 
presents the model fit indices. Based on the outputs, the model was adequately fit with the given data set and 
was valid and acceptable for measurement. 


s* 



Figure 1 . CFA measurement model 
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Table 1. Goodness-of-fit indices for CFA model model fit indices 


CFI 

GFI 

AGFI RMSEA 

SRMR 

Acceptable if > . 900 

>.900 

>.900 <.080 

<.100 

ICTL .970 

.898 

.806 .083 

.042 

Note: (Joreskog & Sorbom, 1993; Hu & Bender, 1999) 



Table 2. Average variance extract and discriminant validity 



Information 

Information 

Classroom 


Seeking 

Sharing 

Learning 

Information Seeking 

0.596 



Information Sharing 

0.814 

0.679 


Classroom Learning 

0.699 

0.618 

0.628 


Note: In the diagonal-running cells, the average variance extracted (AVE) is in bold; the lower-left half of the table shows 
the squared inter-construct correlation estimates (SIC). 


Table 3. Reliability 


Constructs 

Cronbach’s alpha 
(reliability coefficient) 

Information Seeking 

0.82 

Information Sharing 

0.86 

Classroom Learning 

0.89 


All of the Cronbach’s alpha results ranged from 0.82 to 0.89. According to Devellis (1991), these 
reliability estimates can be considered very good. These Cronbach’s alphas suggest that the items in each 
scale were consistent with each other. Convergent validity was examined from the construct reliability and 
the Average Variance Extract (AVE) of each construct (See Table 2). All three constructs demonstrated 
adequate convergent validity with an AVE greater than 0.5. However, the constructs demonstrated poor 
discriminant validity because the information sharing and the information seeking constructs are highly 
correlated. 

3.3 Analysis 

Among the student respondents, 58% of the respondents are male («= 51) and 42% of the respondents are 
female (n= 37). The majority of the respondents are between 15-17 years old (n= 49). When asked about their 
learning behaviors, students reported an average of 3.53 (£D=1.12) of level of agreement on information 
sharing, 3.91 ( SD=.96 ) on information seeking, and 4.05 (SD=. 86) on classroom learning, respectively. These 
constructed variables were the combination of multiple items. The information seeking construct was 
developed using three statements: More classroom learning should use technology tools; I learn more when I 
am free to search for the answers on my own; and I use the Internet to keep current on important topics. The 
information sharing construct was developed using three statements: I learn many things by interacting with 
other Internet users; I like to share information on the Internet with posts and tweets; and I would like to be a 
member of an Internet learning online community. The classroom learning construct was developed using 
four statements: I learn best in the classroom setting; the things I need to know are taught in the classroom; I 
like to take classes to learn new things; and I learn many things in the classroom. 

A Pearson’s Product Moment Correlation analysis revealed significant positive correlations between these 
aspects of information behavior (>=0.672-0.754, p<.00 I ). To answer research question 1, multiple analysis of 
variance (MANOVA) was performed to determine the effects of a categorical independent variable (Gender: 
Male and Female) on three continuous dependent variables including information seeking, information 
sharing, and classroom learning. Although the groups for the male and female respondents were unequal (51 
male students; 37 female students), the Box’s M test (Box’s M=13.869; F= 2.206, /?=(). 04) was found to be 
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non-statistically significant based on Huberty and Petoskey’s guideline (2000) indicating that the 
variance-covariance matrices between the groups were equal for the MANOVA purpose. 

Generally, female students reported having a higher level of agreement on all three aspects of information 
behaviors. The one-way MANOVA on learning preference revealed a significant multivariate effect with 
Wilks’ lambda=.87, F(3,84)=4.18, p=.008, i]2=,130. Approximately 13% of the multivariate variance of the 
learning preference construct was associated with student gender. The univariate effects revealed statistically 
significant differences on classroom learning preference where F( 1,86)=10.36, p=0.002. To be specific, 
female students had significantly higher levels of preference for the classroom learning construct (M=4.38, 
S D=0.65), indicating that their perceptions on classroom learning are more positive than those of male 
students (M=3.81, SD=0.91). See Table 4. 


Table 4. Learning preference by gender 


Learning 

Preference 

n 

Mean 

SD 

F 

Sig. 

Effect Size 
(Cohen’s d) 

Information Seeking 
Male 

51 

3.77 

.98 

2.72 

.103 

0.36 small 

Female 

37 

4.11 

.91 




Information Sharing 
Male 

51 

3.43 

1.08 

1.05 

.309 

0.21 small 

Female 

37 

3.67 

1.18 




Classroom Learning 
Male 

51 

3.81 

.91 

10.36 

.002 

0.72 large 

Female 

37 

4.38 

.65 





The one-way MANOVA revealed a non-significant multivariate effect with Wilks’ lambda=.924, 
F(3,84)=2.29, p=.084, i]2=.076. However, through the examination of the univariate effects, we found that 
there was a statistically significant differences on “classroom learning” preference between STEM and 
non-STEM students, F( 1,86)=6.22, pc. 05. To be specific, students with STEM major demonstrated higher 
level of classroom learning preference (M=4.3, SD=.73) than those of students with non-STEM major 
(M=3.83, SD=.92). See Table 5. 


Table 5. Learning preference by STEM major 


Learning 

Preference 

n 

Mean 

SD 

F 

Sig. 

Effect Size 
(Cohen’s d ) 

Information Seeking 




1.91 

.171 

0.25 small 

Non-STEM 

44 

3.77 

.99 




STEM 

44 

4.01 

.92 




Information Sharing 




3.77 

.056 

0.45 small 

Non-STEM 

44 

3.30 

1.15 




STEM 

44 

3.80 

1.06 




Classroom Learning 




6.22 

.015 

0.53 medium 

Non-STEM 

44 

3.83 

.92 




STEM 

44 

4.27 

.73 





Note: Means are based on ratings from 1 to 5 where 1 =strongly disagree and 5 =strongly agree. 


3.4 Limitations 

Confirmatory factor analysis indicated that while the constructs for information seeking and information 
sharing have strong reliability, they are related, as might be expected for two aspects of information behavior 
and therefore low in discriminant validity. Additional research is planned to determine the extent to which 
additional question items can improve the discriminant validity by reducing the correlation between the 
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information seeking and information sharing constructs. This research is also limited by the sample of 
students who are all from one school community, as well as the constraints of self-reported data. Additional 
research is planned with broader samples of students and observations of actual student information behavior. 


4. DISCUSSION AND CONCLUSION 

Student attitudes towards learning options, such as information seeking in technology-mediated information 
environments, provide valuable information to guide in design of instruction and assist in defining useful 
roles for ICT in informal-to-formal learning. The development of and increased availability of ICT and 
mobile learning options raise questions about students’ tendencies to participate in self-directed learning via 
ICT and also shed light on students’ information behaviors. Students’ preferences for use of ICT tools and 
their information behaviors warrant careful examination because they reveal how students can be expected to 
interact with information in different learning environments. Additional research is needed to explore the 
extent to which learning preferences relate to academic outcomes and self-directed learning activities. 
Validated instrument constructs such as the ICTL information seeking, information sharing, and classroom 
learning scales can improve teaching and learning by providing an understanding of students’ learning 
preferences for the purpose of supporting instruction and informal-to-formal learning. The scales of the ICTL 
were found to be acceptable for use in gauging high school students’ information behavior preferences for 
classroom learning, information seeking, and information sharing. The ICTL constructs, as taxonomy built 
upon empirical data for general, group, and specific factors, may support a theoretical framework for research 
and experimentation (Cattell, 1987) on information behavior. 

Teaching and learning are best facilitated when learning preference can be matched to methods (Keefe, 
1979; Curry, 1981; Owens & Straton, 1980). Students’ information behavior has been related to critical 
thinking skills, which are very important to knowledge construction. Weiler (2005, p. 52) pointed out that we 
assume that students will be effective in seeking information, yet we should be prepared to guide students in 
developing essential information and critical thinking skills. Findings from this study indicated that 
regardless of gender and STEM interest, high school students tend to be positive in their perceptions of 
classroom learning and ICT-mediated information seeking and sharing. As a group, students were most 
positive toward classroom learning, followed by ICT information seeking, then information sharing. The 
standout for the students in this study was the classroom learning construct, which when taken into 
consideration by gender and study major, indicated significant differences of in learning preference between 
groups. To be specific, female students tend to have a higher preference for classroom learning than male 
students. Students with STEM interest tend to have a higher preference for classroom learning than non- 
STEM major students. 
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ABSTRACT 

In the paper, different approaches of process modelling in e-leaming system development are investigated. We provide a 
look at the DIN PAS 1032-1 and in the process model ROME, which is a derivation of the DIN PAS 1032-1. ROME has 
been extended by several pattern approaches. However, after several years of using ROME, we found out that ROME has 
some major flaws, which can be traced back to the so-called top-down approach. In ROME, the decision for a certain 
type of e-leaming system is the first step. After this, the process model is used to structure the development process. In 
reality, in most cases a bottom-up approach would be better suited: centred around the learner and focused on the 
learner’s needs, a content reduction and development should take place, and later, as second level step in the process, a 
decision for a certain e-learning type can take place. 


KEYWORDS 

Process Models, Patterns, Agile Software Engineering 


1. INTRODUCTION 

Coming from the perspective of Computer Science, the domain of e-learning system development is special 
with regards to the combination of stakeholders involved. Like in no other domain, the resulting system is 
depending on how well persons with multidisciplinary backgrounds work together. An optimal team of 
e-learning system developers consists of the psychologist, providing the learning theory background and the 
empirical design, the computer scientist, who is responsible for the system implementation and the system 
architecture, the expert of the application domain, who is responsible for the content development, and who 
has to work hand in hand with the didactic expert, who helps to structure the content in a way that it is 
optimized for learning. The designer is responsible for a human computer interface, which is best suited for 
the target group of learners and for representing the content. And last but not least the learner, who is often 
ignored in the context of e-learning system development, but who plays an important role as target group and 
as beta tester of a learning software. This optimal team can be seen in figure 1, but can be called pure fiction, 
when it comes to e-learning system development. Most of the systems are developed either as research 
artifacts, which focus on special domains (which can be either computer science domains, when a new 
technology shall be tested (e.g. (Martens, Himmelspach, 2005)), or in application domains, e.g. psychology 
(Anderson, 1990)), or as pseudo constructs, when an institution decides that now is the time of e-learning. In 
both cases, the main stakeholder which is the user of the resulting system, i.e. the learner, is completely 
ignored in the process. From the perspective of computer science, e-learning system development usually still 
means to re-invent the wheel, as usually no process models are used (even if they exist since the early 2000s, 
see e.g. (Harrer, 2003), (Harrer, Martens, 2007), (Martens, 2009), (Pawlowski, 2000)). Things go even worse, 
if such an e-learning system is evaluated. Most of the time, evaluation methods are used on a very coarse 
level, chosen investigation periods are too short, no comparable group is given, or the group investigated is 
much too small. 
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Figure 1. Optimal team of e-learning System developers 

With our strong background in instructional sciences, we also have another problem with the state of the 
art in e-learning system development. In instructional settings, the decision process usually has to start with a 
focus on the learner (whom shall learn?), shifts to the content (what shall be learned?). The content then is 
reduced to a level, which meets the learner’s needs (in Germany this is called didactical reduction), is then 
embedded and realized in special purpose methodologies (for education). After all these aspects have been 
taken into account, the decision regarding the multimedia realizations can take place, e.g. using an e-learning 
system. We call this the bottom-up approach, due to the fact that the learning itself shall be the basis of all 
e-learning development, as we perceive e-learning as one of many support technologies to facilitate learning. 
However, over the last years we have observed in most cases a so called top-down approach, which starts 
with the e-learning system, then looks for an appropriate content development and, last but not least, looks at 
the learner. This might be due to the fact that historical e-learning development took exactly this direction 
(Lelouche, 1999 ). However, this can no longer be the state of the art. 

The reminder of the paper is structured as follows: 

In the following section, we give a summary of the existing DIN PAS and ISO/IEC standards in the 
research area of e-learning, as this is the basis of our development. Several years ago, 2006 to 2009, we have 
worked with and extended ROME (Rostocker Model for systematic description of e-learning development), 
which has been developed in the Rostocker Fraunhofer Research Center by Hambach (Hambach, 2008). 
ROME is an extension of the DIN PAS, and has been brought to another level by integrating software 
patterns and the idea of boundary objects. This approach has been recently revisited, as we made the 
experience that even more patterns are usable and that our combined approach has some drawbacks. In the 
conclusion and outlook, we show some additional point for additional work and we sketch our current field 
of research, where we are working to bring agile software methods into process model development for 
e-learning systems. 


2. PROCESS MODELS IN E-LEARNING 

Process models are used a part of software engineering since several years, at least since the early 1990s (see 
for example (Gamma, 1995)). In most cases, these process models have been developed as part of knowledge 
management in multidisciplinary teams, to facilitate communication and to work as boundary objects (see 
e.g. (Martens, 2009)). The general process management models are independent of the context and they are 
applicable to all types of software development projects. However, from the perspective of special 
applications, like for example e-learning, a general process management tool is not easy to use and must be 
adapted to the special team. For example, the aspects instructional design and the empirical design are special 
for e-learning systems and cannot be compared to other system types. 
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As process models for e-learning combine multiple perspectives, which in contrast to the classical 
software developer - customer scenario require communication on different professional levels, it seems that 
a new and special purpose process model is required. Several years ago, this led to the development of the 
system ROME. Since ROME extends the DIN Pas 1032-1 (DIN, 2006), we have taken this as the starting 
point for our current research. 

2.1 DIN PAS and ISO/IEC 

To have a foundation to understand the ROME approach, it is necessary to take a look at the different 
existing standards for a structured development of e-learning proposals. Therefore it is also meaningful to 
consider the chronology of the German and the international development of the different standards. The 
German Institute for Standardization (DIN) published in 2004 the DIN PAS 1032-1 “Learning, Education 
and Training focussing in e-learning” (DIN, 2004). This PAS (Publicly Available Specification) includes a 
reference model for e-learning proposals for all processes of quality, development, implementation and 
evaluation. So the DIN PAS 1032-1 gives the team leader a scheme, on which he can rely during the 
developing, and different use cases. But this reference model is merely focused on the development of 
e-learning systems and the process structure, coming from a comparably technical perspective. Beyond the 
decision for an e-learning system is made before the learning objectives and the learners needs were 
analysed. Thus, the standard cannot be directly transferred to other learning processes and methods, i.e. talks 
between teacher and learner in the classroom. 

After this DIN PAS has been introduced, different other standards were developed. The PAS 1068 
(published in 2006) includes a directly description scheme with the motivation, that different e-learning 
systems could be compared by the components of this scheme (Arnold, 2013). The PAS 1069 (published in 
2009) could be understood as a handbook or another reference model for DIN PAS 1032-1 and works, also as 
the DIN PAS 1032-1, with different examples. As the named standards focus the e-learning system 
development with a good quality management, the PAS 1037 (it’s named in this paper to complete the 
German standards focussing on e-learning) foregrounds the quality management and introduces therefore a 
phase model. The PAS 1037 specifies the communication during the process priority. 

All in all the different standards show, that they are a support for the planner of e-learning system 
developments. As the first standard, the DIN PAS 1032-1, describes the structured developing process, the 
other standards are more or less supplements with examples and concretizations. 

Simultaneous to the first DIN standard in Germany, the first international standard for e-learning concepts 
has been published in 2005. The ISO/IEC 19796:1 describes a process model for the conception of e-learning 
systems in reference to the DIN PAS 1032-1. The ISO standard has been extended in 2009 by the ISO/IEC 
19796:3, which completes the ISO/IEC19796:l by a process model that includes different methods. 

There is also a standard, the so called ISO 9000, which, as does the PAS 1037, focuses the quality 
management and looks secondary to the development of e-learning proposals. There are also some papers, 
which describe the adaption of the ISO/IEC 19796:1 for educational organizations (see e.g. (Pawlowski, 
2007). 

Obviously, all these standards follow the top-down approach, as sketched in the Introduction. This 
explains the publications, transferring the models on the field of education - like the QAM (Quality adaption 
model (Pawlowski, 2007) or ROME (Hambach, 2006). The standards are all starting with the e-learning 
systems and look for a structure or scheme for the development and for an appropriate content development. 
The learner as main stakeholder in the e-learning system development process and the multimedia decision - 
whether we need an e-learning system in the educational process or not - are not discussed. This 
chronological order has to be rearranged. We derived the demand for a model that works bottom-up and will 
not be obsolete, when the planner decides not to use e-learning units. 

2.2 ROME 

ROME is based on the DIN PAS 1032-1, but extends the model by a detailed process description and a 
detailed analysis of phases, tools, roles and artefacts. We will not describe ROME in detail in this paper, but 
give a short overview. As ROME has the same major drawback (the top-down approach), we sketch our 
critique in the following section. 
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As shown in figure 2 on the left hand side, ROME follows the phases: analysis, overall concept 
development, detailed concept development, production, introduction and teaching/learning. The distinction 
of overall concept and detailed concept is one difference to the DIN PAS 1032-1. Each of the phases is 
separated in different steps, which can be seen on the right hand side of figure 2. In the figure, due to 
graphical readability, we have not noted that the sub-steps of each phase might take place in a different order 
or even partly in parallel. Moreover, at the end of each phase, the step manage and evaluate allows for a 
spiral like re-engineering of the sequence of steps. Each of the steps has an associated list of roles, which 
show the responsibility to plan and execute this step. This is an important tool for the project management 
leader, who can use this to structure his team. Each of the steps results in artefacts, which can for example be 
documents, programming parts, data bases or other parts of an e-learning system. In addition to the process 
and step description in figure 2, ROME contains an elaborate role model, where several roles in the 
e-learning system design process are sketched, and an extensive resource model, where a list of potential 
resulting artefacts, their structure and their role in the whole process is given. 



Figure 2. ROME process model for e-learning System development (Hambach, 2008) 

In later works, ROME has been extended at different levels. For example, Harrer and Martens have 
developed a generic pattern language, which has its roots in intelligent tutoring systems and has later been 
refined and extended towards game -based learning and general e-learning systems (see e.g. (Harrer, 2007), 
(Maciuszek, 2011)). 

From our perspective, ROME as a process model has become mid-level pattern, which is used as 
intermediate between other patterns, as can be seen in figure 3. 

The top level pattern would be given by knowledge management patterns. In the context of 
multidisciplinary teams, these are usually communication management and time management techniques. 
One technique, which we like to use as kick off in e-learning project development, is the World Cafe method, 
which can be used to structure conversational processes. In addition we want to focus on the participation of 
all involved during the development process to consider various requirements of the stakeholders. 

The related low level patterns are used by the computer scientists and designers in the team, and they are 
directly related to software development patterns (Gamma, 1995). We have also worked with task models 
and with content patterns from story design (Maciuszek, 2011), which are low level patterns for 
non-computer scientists, like for example story authors in role playing games. 
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Figure 3. Levels of Patterns in Process Modelling 


3. RE-ENGINEERING ROME 

After using ROME for several years now, we have experienced the following drawbacks: even if ROME 
allows interpreting some steps as responsible for the instructional design (e.g. step C2 and D2 have been used 
for this purpose), an explicit integration of instructional design (or didactical design, as we prefer to call it in 
Germany) is necessary. Accordingly, after integrating instructional design, the design process as a whole is 
not any longer correct. If we are thinking in terms of instructional design, the decision for certain content and 
the reduction of this content to the educational purpose is missing as prior decisions. Thus, the process 
sequence analysis - concept - production is not valid any more. In ROME, a classical top-down approach is 
used, where the first step is to decide that an e-learning system shall be developed. The look at the person in 
the centre, the learner, is taking place comparably late in the process (see figure 2), after the educational 
setting is analysed. 

In our bottom up approach, we use the sequence learner analysis, content analysis, content reduction, 
instructional concept, technical concept, evaluation concept and production. Moreover, also these phases can 
be re-visited in a spiral manner. We suggest starting with a specification of the target group (who is the 
learner and where should he learn?), followed by the analysis of the content (what shall be learned?). A very 
important step in this context, and flaw of quite a lot teaching and training approaches, is the missing 
instructional reduction of the content. This means that the content itself is not adapted to the target group, to 
the prior knowledge of the target group, to the cognitive level (or the assumed level), etc. After the reduction 
of the content took place, the instructional methods used for re-structuring the content in an instructionally 
adequate way must be chosen and the design of the content can happen. This, again, has to take the learner 
into account. Additionally, information about the learning setting must influence this design. After all these 
analysis and design steps took place, the decision about the e-learning system itself can be performed. A 
decision here can for example mean to decide for a certain form of e-learning (e.g. intelligent tutoring, 
game-based learning, simulation, etc). After that, the communicative process about how to bring content and 
computer system together can take place. However, at this point of development, this is mainly a technical, or 
software engineering decision. 

Interesting in our approach is also the fact, that the usage of e-learning can be completely excluded: the 
design of an instructional learner centred process can lead to the decision to explicitly not use a computer in 
the teaching and training of certain content. Naturally, this is quire depressing for e-learning system 
developers, as we are, but we think that the idea to open up to this direction helps us to develop really useful 
and usable e-learning systems. 

The next aspect, which is completely missing in ROME, is the design of the evaluation of the e-learning 
systems. In the last years, we found out, that so called empirical evaluation of e-learning is in most cases only 
on a very bad level: usually, hypotheses are missing; in most cases, the compared groups are not comparable, 
as instructional design varies between computer based and non-computer based setting; and in most cases, 
the evaluation is only used as a vehicle for justification of just another e-learning system. 

Although another aspect is the knowledge management during the development. Different management 
techniques that focusses on participation of all involved stakeholders and quality management should not be 
excluded. 
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4. CONCLUSION AND OUTLOOK 

Most of the e-learning systems, which can be found in research and even in commercial settings, are 
developed off scratch. This is surprising, as e-learning system development comes with comparably high 
costs, is very time consuming, and. on the other hand, lends itself for structured approaches. Moreover, 
diverse structured approaches exist since the early 2000s. As a small overview over the diverse approaches, 
we have focused in this paper our own work, which has started from the DIN PAS 1032-1, refined this 
towards the ROME model, which has been used as centre point for multi-level pattern developments. We 
have sketched these approaches in the section 2. However, after several years of using ROME, we found out, 
that our approach has some major drawbacks. Thus, we have decided to re-design ROME. 

As ROME starts with the idea to develop an e-learning system and then takes into account learners and 
content (which is called top-down approach in this paper), we have decided to start at the roots: as the learner 
is the centre of each learning process, we develop a bottom-up approach, which is based on an analysis of the 
learner, and analysis and reduction of the content and an adaptation of the content to the learner. 

Our current revision process of the process model ROME and the related patterns lead us to the idea to 
combine this with agile software development methods. 

In order to adopt the idea and the principals of agile software development for learning processes, we will 
discuss the fundamentals of agile software development. Based on the “agile manifest” we accentuate the 
aspects communication, collaboration and flexibility (Beedle, 2001). These notes illustrate a continuity of 
feedback polish between all included persons and the possibility to adapt the product or the process during 
the development. 

Remembering on the painted experience of actual developing processes, we could repeat the request of 
the following description: It is necessary to work hand in hand with the person, who is responsible for the 
content development, the didactic expert, the designer and the learner. 
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ABSTRACT 

The possibility of using serious games with problem-based learning opens up huge opportunities to connect the 
experiences of daily life of students with learning. In this context, this article presents a model for serious and ubiquitous 
games development, focusing on problem based learning methodology. The model allows teachers to create games based 
on problems. Besides, such games can be adapted to real-world scenarios, using context-aware information to enable 
interaction with real situations in that environment. The main scientific contribution of this work is related to teachers’ 
autonomy, creating a motivating learning environment and fostering the relationship between theory and practice. To 
evaluate the model, we developed a use case, detailing the processes of game creation by the teacher, in an adaptive 
environment, and its use by the student. As a result, we expect greater student engagement in the proposed activities and 
to promote the development of higher order skills. 
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1. INTRODUCTION 

The spread of information and communication technologies is present and influences the social and 
professional life. Thus, there is no way to separate knowledge regarding Information and Communication 
Technologies (ICTs) from other areas of human knowledge. According to Carvalho (2011) "society and 
technology are inseparable phenomena and the changes that occur in one of them changes the other." This 
implies a new form of language and communication, which is perceptible nowadays in the consolidation of 
digital mediation, which must be applied to teach and learn. Holland and Holland (2014) reinforces that new 
technologies offer a great way to invigorate instruction, whether in traditional classrooms, online, or in 
blended learning environments. 

As a result of this need, we observe an increasing use of methods to assist in the learning process, using 
the most assorted technologies. Thus, theories and learning methods that encourage and motivate students 
with the use of technologies gain strength, among them stand out the games and educational software (NETO 
and SOUZA, 2012). 

The use of games to encourage motivation and assist in cognitive development is hardly a novelty 
(RIEBER. 1996), (GARRIS et al., 2002). However, the evolution and the combination of mobile and 
ubiquitous technologies allow expansion of the game into the real world (SEGATTO et al, 2008), (CHEN 
and HUANG, 2012), thus allowing space, time, and social relations to be part and interact with the rules of 
the game, permitting to expand student interaction with the object of study, regarding context aspects. 

This article proposes a model for the development of serious games, called UCHALLENGE, fostering 
motivation in the field of education. This model uses ubiquitous resources as a way of interaction, thus 
enabling a real environment to be part and an extension for a playful learning. Associated with this 
technology, a pedagogical proposal is used, supported by the Problem Based Learning (PBL) methodology 
(BARROWS, 1980). With this, we hope to expand the autonomy of the student, through features and 
challenges, propose the construction of solutions to real or simulated problems, making them a motivator for 
learning, thus mobilizing different areas of knowledge. 
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The possibilities of expanding a game from the virtual to the real world, not only as a fun factor, but also 
with well-defined educational objectives, present new opportunities for interaction between knowing and 
doing. However, we are not referring to any game, but those that focus on learning purposes, the so-called 
serious games (CLARK. 1970; GIESSEN, 2015), whose main purpose is not only fun, but also Uaining, 
simulation, marketing or education. 

The expansion of the game into the real world is based on the possibilities of ubiquitous computing, as 
defined by Weiser (1991) and reinforced by Satyanarayanan (2001). The latter defines this concept as the 
"creation of environments saturated with computing and communication capabilities integrating them with 
users", allowing also the possibility to link and extend learning content with real situations of everyday life. 

However, for this to be effective, it is important to create conducive conditions to the student in this 
autonomy process and search for knowledge. Therefore, we need to use learning methods that assist in the 
knowledge building, associated with information technology. 

In the above-mentioned learning theories, which are presented to us, the question that arises is how to find 
a conceptual and general epistemological model that incorporates technology to learning ? Several attempts 
of using educational games (SANCHES and OLIVARES, 2011), (NETO, 2012), (MORAN, 2012) have been 
tried as a proposal to assist in this process. However, some games end up emphasizing traditional repetition 
methods with a technological guise. Therefore, we must find ways to harness the motivation provided by the 
games to assist students in their knowledge building and autonomy, connecting the virtual and real world. 

In this sense, the problem-based learning, hitherto restricted to specific segment of Medicine, has gained 
strength in schools, colleges and universities in different areas of learning (SAVERY, 2006). When using the 
problem based learning method, students are encouraged to think critically and they are given responsibility 
in building their knowledge (OZBI(TAK£L et al., 2012). Thus, learning is directed by the student in small 
groups, through problems, which are systematically resolved to achieve the intended educational goals, and 
seek autonomy, work, and cooperate in team skills. 

Barrows (1980) points out that learning from problems is a condition of human existence and that 
learning occurs in our attempts to solve the many problems we face every day. Therefore, approaching and 
encouraging students through challenges in solving problems can significantly help in the process of learning 
autonomy. A serious game, using ubiquitous resources, can motivate and encourage the knowledge building, 
allowing for even more interaction between the student and the context, thus providing opportunities for 
more action in this process. According to Piaget (1967) cited by Clark (1974, p.ll) "Intelligence is born of 
action, and anything is understood only to the extent that is reinvented." 

Consequently, the goal of this work is to specify and evaluate a model for creating educational serious 
games that use resources of ubiquitous computing, using a methodology based on PBL. As a result, we 
expect the model to provide greater motivation for the student, thereby creating new learning opportunities. 
The main scientific contribution of this work is to enable the teacher to create a real learning environment 
that fosters the interaction between theory and practice. 

This article is organized into five sections. Section two presents related works. Section three discusses the 
proposed model in details. A case study showing teacher and student role is presented in section four. Finally, 
we discuss the conclusions and directions for future work. 


2. RELATED WORK 

Several studies address serious games, using concepts of mobile and ubiquitous computing. As the target of 
this study, we selected models that jointly dealt with the issues and had, as main point, learning goals. 

Huizinga (2008) points out that "every game is able at any time to fully absorb the player" setting for that 
social, temporal, and spatial aspects. The use of ubiquitous games creates the possibility of expanding these 
aspects, thus increasing the contractual magic circle of a game (MONTOLA, 2005), creating new social, 
spatial or temporal relations beyond the virtual world, facilitating the expansion to the real world. The 
authors also suggest ways and use of technologies for creating ubiquitous games for educational purposes. 

Chang et al, (2009) presents an educational game, supported by mobile technologies. In the proposed 
model, they employ location, geographic information systems (GIS), and sensors such as Radio Frequency 
Identification (RFID), to develop a ubiquitous learning environment. The aim of the proposal is to improve 
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the students' participation in the planned activities, targeting to study a course whose learning area covers 
Taiwan culture. 

Laine et al., (2010) reports studies performed within a period of three years by the team at the Ubiquelab 
from Eastern Finland University. The study discusses possible uses of technology in school, through a model 
for games based on pervasive learning spaces (Pervasive Learning Spaces - PLS). Experiments are laid out 
based on the development of ubiquitous games using a hipercontextualized design model, that is, "where the 
game is rooted in the same context in which the player is embedded" (LAINE et al., 2010). 

Liu and Chu (2010) present the results of a game and its ubiquitous possibility targeting English learning, 
and also motivation in that regard. The model, called HELLO, integrates cameras and Quick Response Codes 
(QRcodes) tags distributed in the learning environment, allowing the student interaction in the available 
contexts. 

Weatherlings (KLOPFER et al. 2012), is a collectible strategy game card, similar to several other games 
(Trading Card Games - TCG). In this game, the user assembles a custom deck, combining strategic battles, to 
play against other opponents. The deck is composed of weather creatures and the information available to 
students is composed of sources of weather data. The activity allows the interaction of students with real data 
regarding climate and weather. According to Klopfer et al. (2012), the game aims to provide users to "be able 
to read successfully weather tables to plan appropriate strategies to actual historical weather data." 

Chen et al. (2012) presents a model and results of a Ubiquitous system called adventure game (RPG) 
learning system based on the game. The authors aim to improve the learning experience through ubiquitous 
games. Thus when students are doing the learning activities, the system will record the path of learning 
through GPS device, text files and progress. When the game ends, the system shows the learning progress 
and related content. 

All models considered have some ubiquitous characteristics, which can be understood as "the 
coordination of intelligent devices, mobile and stationary, to provide users immediate, and universal access to 
information and new services transparently, to increase human capabilities" (SANTAELLA, 2013). Hence, 
considering these related works, one can consider that some aspects such as the method of learning and 
motivation from the real world to the virtual intersection can be better exploited. 

The models presented do not allow the creation and adaptation of the game by the teacher and there is 
neither ontology that manages the steps of the proposed challenges, nor adaptation scenarios that can be used 
in different areas of knowledge. 


3. PROPOSED MODEL 

The proposed model, called UCHALLENGE, seeks to provide a possibility of creating serious games 
supported by the PBL methodology. In the proposal, the teacher can manage the game by setting the domain 
to be used, content, problems, and challenges that will be available at each stage of learning. Other 
characteristics that the teacher manages include the scenario and the ubiquitous resources that can be 
integrated in the context in which the game will take place. 

UCHALLENGE is based on three aspects: ubiquitous computing (WEISER. 1991), 

(SATYANARAYANAN, 2001), serious games (CLARK, 1974), and problem-based learning (BARROWS, 
1980). Thus, the following aspects take part of the model: 

• to be adaptable to different domains, allowing the teacher to organize the game according to the area of 
interest; 

• to allow the insertion and use of information: the teacher has the possibility to manage the database as 
needed, making use and reuse of existing data; 

• to provide the creation of problems and challenges: to enable the teacher to develop problems and 
sub-problems and the challenges that will be part of each stage of the game; 

• to provide environment information: to enable the teacher to generate challenges that can be adapted to 
the context, defining the points of interaction; 

• to manage problems and challenges, so that the student can receive the challenges in each stage of the 
game and level of learning; 
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• to use an ontology, based on Bloom taxonomy (Bloom, 1956 apud Chan et al., 2009) and its cognitive 
objectives, providing challenges, content and problems according to context and development of the student, 
taking into account an increasing complexity; 

• to store game results, including paths followed by the student during the game; 

• to manage the game, enabling students to start a game or resume where it left off before; 

• to support ubiquitous resources and contexts, including QRcode , RFID sensors and geolocation 
systems; 

• to allow the interaction between users and teachers, providing communication between users to discuss 
and solve game challenges, as well as interventions and feedbacks; 

• to foster social integration, enabling access to social networks for exchange and dissemination of 
information; 

• to be attractive, fun and motivate the student, so that the user want to continue playing. 

Figure 1 presents the UCHALLENGE model. Generally, the architecture is divided in two components: 
teacher and student modules. 



Figure 1. UCHALLENGE Model 

The tutor module initially presents a learning management interface, in which teachers can access and 
manage the game, content, problems and challenges that will be used. Moreover, in the tutor module, the 
teacher can manage the map of the scene, defining the points of interaction and the rules of the game, as well 
as the feedback to students. The points of interaction and environmental challenges are defined on the map of 
the scene and converted to sensors that will be distributed, as defined by the tutor. 

The student module displays the home interface, wherein the available games are displayed. When 
selecting the game, students receive the first step, or, if they had already started, obtain the corresponding 
step, and the scenario with the possible points of interaction. The management of the game controls the rules, 
history, access to the base of the student and content, besides enabling the use of ubiquitous resources 
available, as geographical location in relation to the scenario, communication with the available sensors and 
points of interaction. 

The selection of challenges and problems offered to the student, are defined using an ontology, as in 
Figure 2. The ontology is based on the educational objectives defined by the teacher, from the cognitive 
processes established by Bloom (1956) and reinforced by Chan et al., (2009). Thus, levels of increasing 
difficulty will be presented to the student representing an evolution in their learning processes. 
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Figure 2. Relationship of the main classes of the ontology 


4. CASE STUDY 

To evaluate the UCHALLENGE model, we developed a case study to analyze usage scenarios in relation to 
the teacher and the student interaction. The scientific community has been using scenarios to evaluate mobile 
and ubiquitous computing projects (DEY, 2001; SATYANARAYANAN, 2001 and SATYANARAYANAN, 
201 1 ). Therefore, we proposed a scenario for a lesson in a History course. This scenario was evaluated with a 
developed prototype. The teacher's web interface has been developed in PHP language and Javascript using 
MYSQL database. For the student interface, we used the Android platform and developed an application 
using HTML5. 

A Phstory teacher has been teaching Global History, particularly Middle Ages, with the class. Upon 
bringing the subject on the early 11 th Century, comes the need for students to make reflections on what 
aspects led to the Crusades and how it is possible for them to relate this concept to the present day. In order 
to encourage students to better learn the historic facts regarding the Crusades, the teacher will make use of 
UCHALLENGE to build a game set in a park, which simulates medieval forests. With the access to the 
server through the Web, the teacher logs on into the learning management interface. 

In this interface information concerning the game are defined, such as: Area of knowledge: "Human 
Sciences"; Assessment area: "Plistory"; Theme and the game: “Crusades”. The teacher defines the game 
scenario, from a map, which obtains the geographic coordinates of the specific park (Figure 3). The student 
then loads this piece of information. The teacher also registers the main problem to be solved: Why was the 
Holy Land the cause of many conflicts between Muslims and Christians? This conflict still exists today? 

In the UCPIALLENGE environment, the teacher registers also learning objects that will be available to 
the student’s research, as well as, problems and challenges. By registering the problems, the teacher needs 
also to define sub problems to be presented to the students in different stages of the game, as the following 
example: 
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Figure 3. New game development: Selecting the place 

• What happened between 1 1 th and 13 th Centuries in Europe? 

• Do you know what the Crusades were? 

• What were the reasons that led to the formation of the First Crusade? 

• Which countries were involved in the battles and what were their real objectives? 

• Which issues are still reasons for dispute today? 

The contents, problems and challenges available are uploaded from the content base, in which they are 
identified by area of knowledge, assessment area, subject and pedagogical goals, so that they may be inferred 
by the ontology and presented to the student as the progress of the game. 

Teachers prepare the environment of the game, selecting through a specific tool, the map coordinates with 
the game scenario information. In this defined scenario, the teacher will use QRCode and RFID sensors to 
create points of interaction with the player. These sensors contain specific clues to solve the problems or 
challenges during interaction with the context. The created points of interaction are displayed on the map of 
the environment and loaded later, from the player's position. After finishing the game settings, the teacher 
physically makes the sensors available in the game scenario, releasing the environment for students. 

The student will have to register in advance in the system, after authentication (Figure 4a) the new games 
or the games in progress will be showed. After the game selection, students can access information about 
other players who are in the same game and may interact to each other to solve challenges or proposed 
problems. The interaction may be through messages or contact through social networks, if there is such a 
possibility. 

The student, after registering, connects with the online server via a mobile device, which shows the 
available games (Figure 4b). Once the game starts, the student receives the main question, as well as a 
number of small problems on the subject (Figure 4c). The path chosen by the student, within the proposed 
scenario, will define what challenges and problems will be offered. The game map (Figure 4d) presents its 
geographical position in the scenario and possible interactions. Upon arriving at the point of interaction, the 
student drives the marker, thus, the QRcode reader in the mobile App is enabled to capture the information 
available on the tag. Such information may have practical or theoretical challenges to be solved in the 
environment. Consequently, the result of the challenge ensures to the student points or important tips for the 
next steps. At any time of the game, one student can share information with the others, through messages 
related to the challenges posed. 

The teacher defines the rules of access to learning objects. They may be, for example, objects in text or 
multimedia format accessed freely for consultation. Alternatively, its is possible to specify that some content 
costs a number of knowledge coins, which are acquired through the solution of a Quiz, challenges, or extra 
missions that are presented in the game scenario. 
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Figure 4. Initial Player Screens 

In addition to the knowledge coins, the player also receives a score for each correct answer. Each time the 
student reaches 100 points, it can answer the problem proposed to that step, enabling the student to move to 
the next level. 

The problems and challenges presented to the student are inferred in the ontology, through pedagogical 
objectives previously registered by the teacher, based on Bloom taxonomy of increasing complexity. After 
passing through all steps in the scenario, the student must still solve the main problem. The student may 
make use of the knowledge and experiences acquired during the game to present a possible solution that will 
be evaluated by the teacher. 

At the end of the game, the student receives feedback from the teacher and share experiences and doubts, 
enabling new reflections on the working theme and analysis of the results obtained. From these results, the 
teacher may decide to review a subject or to create new possibilities to work with the class. 


5. CONCLUSION 

The UCHALLENGE has shown to be a viable option for building serious and ubiquitous games focusing on 
problem-based learning. The main advantage of the model is the real possibility of motivation of a student for 
a type of learning able to mobilize a set of cognitive resources and apply different knowledge using Problem 
Based Learning. 

The proposed model presents, as the major scientific contribution, the viability of adapting different 
scenarios and areas of knowledge in a proposal of reflection on theory and practice by the means of solving 
problems. Unlike this proposal, the related works considered for this paper focus on specific approaches and 
with limited use of ubiquitous resources. A case study showed the viability of the model for preparing a class 
combining the real and virtual world. 

As future work, we intend to review the creation of the game in a pedagogical perspective and the 
usability in different areas of knowledge. Besides the evaluation of the student in a real scenario, we plan to 
analyze aspects of motivation and engagement in the activities proposed, as well as its reflection in the 
students learning. 


153 



ISBN: 978-989-8533-43-2 ©2015 


REFERENCES 

Barrows. H. S. 1980. Problem-based learning: An approach to medical education. Springer Publishing Company. New 
York, USA. 

Carvalho, M. G. D. 2011. Tecnologia, desenvolvimento social e educagao tecnologica. Revista Educagao & Tecnologia, 
No 1. 

Chang, W. C. et al, 2009. Game-based learning with ubiquitous technologies. Internet Computing , IEEE, vol. 13, No. 4, 
pp. 26-33, 

Chen, J. H. et al, 2012. To develop the ubiquitous adventure RPG (role play game) game-based learning system. 
In Systems, Man, and Cybernetics, IEEE International Conference on. Seoul, Corea, pp. 2973-2978. 

Clark C. Jogos simulados: estrategia e tomada de decisdo. J. Olympio, Rio de Janeiro, Br. 

Dey, A. K. 2001. Understanding and using context. Personal and ubiquitous computing , vol. 5, No. 1, pp. 4-7. 

Garris, Rosemary. Et al, 2002. Games, motivation, and learning: A research and practice model. Simulation & gaming, 
Vol. 33, No. 4, pp. 441-467. 

Giessen, H. W. 2015. "Serious Games Effects: An Overview." Procedia-Social and Behavioral Sciences, Vol. 174, pp. 
2240-2244. 

Holland, J., & Holland, J. 2014. "Implications of shifting technology in education." TecliTrends Vol. 58, No. 3, 
pp. 16-25. 

Huizinga, Johan. 2008. Home ludens: o jogo como elemento de cultura. Perspectiva, Sao Paulo, Br. 

Klopfer, Eric et al, 2012. Ubiquitous games for learning (UbiqGames): Weatherlings, a worked example. Journal of 
Computer Assisted Learning, Vol. 28, No. 5, pp. 465-476. 

Laine, Teemu Henrikki et al, 2010. Critical factors for technology integration in game-based pervasive learning spaces. 
Learning Technologies, IEEE Transactions on, Vol. 3, No. 4, pp. 294-306. 

Liu, Tsung-Yu and CHU, Yu-Ling. 2010. Using ubiquitous games in an English listening and speaking course: Impact on 
learning outcomes and motivation. Computers & Education, Vol. 55, No. 2, pp. 630-643. 

Montola, M. 2005. Exploring the edge of the magic circle: Defining pervasive games. In Proceedings ofDAC, Vol. 1966, 
pp. 103. 

Moran, J. M. et al. 2012. Novas Tecnologias e mediagao pedagogica. Papims, Campinas, SP, Br. 

Neto, J. F. B. and Da Fonseca, F. S. 2013. Jogos educativos em dispositivos moveis como auxllio ao ensino da 
matematica. RENOTE, Vol. 11, No. 1. 

Neto, J. F„ & Souza, F. D. F. D. O. 2012. Uma Metodologia de Desenvolvimento de Jogos Educativos em Dispositivos 
Moveis para Ambientes Virtuais de Aprendizagem. UFPE, Recife, Br. 

Ozbigakgi, $. et al, 2012. Assessment of goals in problem-based learning. Nurse education today, Vol. 32, No. 8, 
pp. e79-e82. 

Rieber, L. 1996 Seriously considering play: Designing interactive learning environments based on the blending of 
microworlds, simulations, and games. Educational technology research and development, Vol. 44. No. 2, pp. 43-58. 

Sanchez, J., & Olivares, R. 2011. Problem solving and collaboration using mobile serious games. Computers & 
Education, Vol. 57, No. 3, pp.1943-1952. 

Santaella. L. 2013. Comunicagao ubi'qua: repercussoes na cultura e na educagao. Ed. Paulus, Sao Paulo, Br. 

Satyanarayanan, M. 2001. Pervasive computing: Vision and challenges. Personal Communications, IEEE, Vol.S, No. 4. 
pp, 10-17. 

Satyanarayanan, M. 2011. Mobile computing: the next decad t.ACM SIGMOBILE Mobile Computing and 
Communications Review, Vol. 15 No. 2, pp 2-10. 

Savery, J. R. 2006. Overview of problem-based learning: Definitions and distinctions. Interdisciplinary Journal of 
Problem-based Learning, Vol. 1, No. 1, p. 3. 

Segatto, W. et al, 2008 Mobio threat: A mobile game based on the integration of wireless technologies. Computers in 
Entertainment (CIE), Vol. 6, No. 3, p. 39. 

Weisser, M. 1991. The computer for the twenty-first century. Scientific American, Vol. 265, No. 3, pp, 94-104. 


154 



12th International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2015) 


BLENDED INTERACTION FOR AUGMENTED 
LEARNING-AN ASSISTIVE TOOL FOR COGNITIVE 

DISABILITY 


Suman Deb 1 , Subir Saha 1 and Paritosh Bhattacharya 2 

1 Computer Science and Engineering Department 
~ Mathematics Department 
NIT Agartala, India 


ABSTRACT 

The fundamental right of education and employment for cognitive disable person is recognized in very recent times. With 
the advent of technology there have been many interactive way to combat with the challenges of disability and provide 
supplement training for enabling the challenged person with the skill of employability, so that they can individually lead 
the normal life without depending on others. In this work we try to identify the problem faced by cognitively week people 
and find appropriate solution by introducing computer aided interaction. We proposed a platform to blend audio and 
depth vision for augmented learning. The primary goal of this work is to create a play way teaching environment for 
individuals with learning disability so that the enhanced method help them for elementary education and provide 
self-confidence and motivation and encourage for continuous learning. While implementing, the system has shown 
significant improvement on learning by a range of people including learning challenged and normal kinder garden kids. 
The augmented method proposed here can be added advantage with the concept of visual teaching, live experience and 
reactive and pro-active mechanism rather than reading static content from book. This will also involve the physical 
movement with precise control that can lead to motivated control of limbs by any neuro motor disability effected person. 
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1. INTRODUCTION 

A cognitive disable child, besides processing a low I.Q, demonstrates impaired or deficient adaptive 
behaviour originating from conception and continuing into maturity. They fail to learn what average children 
learn and find difficult to detect an absurdity in a logical statement. They are also limited with respect to 
imagination. They remain behind their classmates and become bitter and hostile towards them and others. 
Repeated failures deprive them of confidence and lack the motivation to learn. In a highly technological age, 
where talent are very much needed, dissipation of human resources is great problem .We can hardly ignore 
the gravity of the problem and the care education and training of these children are of great importance and 
significance. Main deficits that people with cognitive disabilities demonstrate are: 

A. Memory: Memory refers to the ability to be able to recall what has been learned over time. Meaningful 
information is typically moved from sensory memory (stored for seconds) to working memory (stored 
minutes) and then stored in long term memory. People with cognitive disabilities have difficulty with one of 
these types of memory [1], 

B. Problem-solving and attention : People with cognitive disabilities often have difficulty problem 
solving. One difficult problem arise, such as learning new material in class, they can typically become 
frustrated and have difficulty expressing their frustration and have difficulty focusing on the task [1], 

If we teach the children who faces some kind of learning disability, in more interactively and visualize the 
teaching content as natural environment rather static content, they can find interest in learning and develop 
feelings of power to overcome difficulties and accumulate a sense of self satisfaction. So far many interactive 
systems have been developed but very few interactive systems were developed to educate them based on 
what has been taught in the class room. Interactive learning provides keen interest in learning and makes 
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learning fun therefore they can remember things more easily. The main focus of this paper is to develop an 
interactive learning platform for assisting them to learn more quickly through playing and visualization. 
Microsoft Kinect sensor is used for making the system. It contains one IR sensor and IR emitter to process 
depth data. It will return depth data of a point with 16 bit gray scale format with a viewable range of 43 
degrees vertical and 57 degrees horizontal [2], The depth data contain the distance between the device and 
the object in front of the device. For example, if a pixel coordinate is 200 x 300, the depth data for that pixel 
point contains distance in millimeters from the Kinect device .figure [1] shows the depth data processing of 
the Kinect device. 



I Distance m mMlwnetre ) 



Figure 1. Depth Data Processing [2] of Kinect Sensor 

In this proposed prototype Kinect Microphone Array and Kinect speech recognition engine are also used 
to recognize the speech. Advantages of using Kinect sensor is that it provide us 20 point human skeleton [2] 
through which gesture can be easily recognized. Moreover the Kinect microphone array have the capability 
to Identify the source direction of the incoming sound and helps to recognize human speech very clearly by 
focusing only in a particular direction and cancelling noises in the environment. Figure [2] shows the 20 
point skeleton joint of human body return by the kinect sensor. 



Figure 2. 20 point Skeleton joint [2] of human body 


2. EXISTING WORK 

Interactive building blocks [5] were developed for the children to learn the concept of geometric structure 
and shape. First time the system deployed a picture and gives some instruction to build the object. A pattern 
recognition algorithm is used to compare the children assemble object with the displayed picture .Virtual 
Laboratory[6 1 has been developed by using kinect Unity 3D and gesture classification algorithm which is 
mainly used to assist students to gain interest on particular subject. In paper [7] author developed a useful 
interactive tool for learning math using blender and Kinect to make learning more visual animated and lively. 
In paper [8] author developed a e-learning system where a student in remote location effectively interact with 
the professor by hand gesture. Therefore professor can pay equal attention to both the remote and local 
student. In paper [9] author proposed an interactive learning platform which combine the full body motion 
sensing a virtual environment. Therefore student can enter into the virtual environment and have interaction 
with the object and the virtual character gives some response to this student. 
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3. SYSTEM ARCHITECTURE 
3.1 Hardware and Software System 

Kinect was launched on 4 November 2010 by Microsoft and built specifically for the Xbox 360 gaming 
console. It enables the user to interact directly with the Xbox, allowing the user to perform touch -free 
operation. The key components of Kinect are ( 1 ) Multi-array microphone, (2) IR laser emitter, (3) IR camera, 
(4) Motorized tilt, (5) USB cable and (6) RGB camera .Below figure [2] shows the key component of the 
Kinect sensor. Microsoft visual studio 2010, C#, kinect for windows SDK and in back end Microsoft Sql 
server 2008 is used to develop the system. 


2 3 



Figure 3. Block diagram of Microsoft Kinect Sensor [3] 


3.2 Proposed System 


In this work we created a platform where the sensor take the depth information of the object and send it to the 
process box. This process box performs depth identification .depth analysis and depth derivation , after 
analyzing the depth data the computer create their own vocabulary and make a mapping of that vocabulary 
and store it for future use . Over view of our proposed system are shown in figure [4], 



Figure 4. System Architecture of the Proposed System 

We divide the system into two phases. 

A. Encoding phase 

B. Recall phase. 

A. Encoding Phase 

In this phase when a child comes in front of the Kinect sensor it will first extract the skeleton by 
processing the depth information and identifies the hand joint. Through this it will recognize the hand portion 
of the children, after that when the children moves his/her hand in front of the sensor it will map the hand 
point and draw the object into the computer screen according to the hand movement. After drawing the object 
he/she assign any pre defined name of this drawing object. The Kinect sensor first recognize the assign name 
using the Kinect speech recognition engine and mapped the drawing object with given name and store the 
mapped information into the database. This step was performed repeatedly for acquire accuracy in decision 
making. Work flow diagram of encoding phase is shown in figure [5]. 
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Figure 5. Workflow diagram of encoding phase. 

B. Recall Phase 

In this phase when a child says any word in front of the sensor, it starts its speech recognition engine for 
recognising the word. If the word is successfully recognised then the system will try to find the mapping 
between the word and the image from the database. If mapping is found then the image corresponding to the 
recognized word is displayed in the screen. Below figure [6] shows the workflow diagram of the recall 
model. 



3.3 Algorithm Used 

3.3.1 Algorithm for Hand Portion Recognition 

Position of both hand for making interactive learning are acquired after skeleton analysis of depth image data 
return from the Kinect sensor. Following algorithmic step was performed to detecting Hand portion. 

Step 1: Capture the x,y,z coordinate of all 20 skeleton Joint 
Step 2: Analyse the x,y coordinate of all joint 
Step 3: Compare the coordinate as 

If(x,y coordinates around hand) j 
If( x,y coordinates around Hand_left) { 

Left hand is recognized. 

} ' 

Else { 

Right Hand is recognized. 

} 

Else l 

Hand is not recognized go to Step 2. 

} 

Step 4: stop 


158 



12th International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2015) 


3.3.2 Algorithm for Speech Recognition 

Recognizing the Speech we are used Kinect Speech recognition Engine. The speech recognition engine 
consists of the following two major modules [2]: 

• Acoustic model 

• Language model 

Each one of the modules has a sole responsibility for recognizing speech. The following is the list of 
operations performed for recognizing the speech: 

Step 1: Microphones capture the audio stream convert the analog audio data into a digital wave. 
Step 2 : The audio sound signals are sent to the speech recognition engine to recognize the audio. 
Step 3: The acoustic model of the speech recognition engine analyzes the audio and converts the 
sound into a number of basic speech elements; we call them phonemes. 

Step 4: The language model analyzes the content of the speech and tries to match the word by 
combining the phonemes within an inbuilt digital dictionary as. 

If (the word exists in the dictionary) { 

Recognize the word 
j Else { 

Word is not recognized. 

} 

Step 5: Stop. 


4. MACHINE LEARNING USING COGNITIVE MAP 


The word cognitive map refers to a process of mental activities which abstract the information from a real 
world scenario and encoded into the memory for automatic recall [4], It represents the cause - effect 
relationships of event in knowledge. Modelling of cognitive map using fuzzy logic is called fuzzy cognitive 
map. It has been recently been introduced in the field of machine intelligence coined by Bari kosko [4] .It is a 
graph structure capable of encoding knowledge. In this work we used Pal and Konar’s Fuzzy cognitive model 
(FCM). In this approach they represent cognitive map as an associative structure consisting of nodes and 
directed arcs where the nodes carry the fuzzy belief and the arcs or edge carry the connectivity strength 
[4]. Here we represent each recognized word and the image associated to the word as a node and the edge 
between the nodes represent the connectivity strength. Below figure [7] shows the cognitive map of the 
proposed system. 


Wj:=0.4 


Wjj=0.3 
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Figure 7. Cognitive map of our proposed system. 

It follows the encoding and recall cycle to build the cognitive map. Process of creating fuzzy cognitive 
map of our system is as follows. 
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Let ni, nj be the fuzzy belief of node Ni and Nj. 

Stepl: Represent each recognised word and drawing object as node. 

Step 2: follow the Hebbian learning to find self mortality ofWij as: 
dWij (t) =a Wij (t) +5 1 (ni (t)).S (nj (t)). Where 
S (nk) =1/ [exp (-nk)J for k= [i, jj. a represent the mortality rate 
Step 3: determine the recall model as 

ni (t+1) =Max [ni(t),Maxk[(nk Min wki)j] 

Step 4: repeal step 2 and 3 (Encode-Recall Cycle) until steady state condition is reached. 
Step 5: Stop 


5 . EXPERIMENT AND RESULT. 

For measuring the accuracy, we test the proposed system several times and found that the performance of 
encoding and recall cycle is satisfactory. Following figure shows some of screen shot while testing the 
system. 




Figure 9. Screen shot while testing recall phase of the system 

The results in Tables show that in the majority of the attempts are successfully recognized and the 
corresponding object are displayed in the screen. This system is tested on different noisy environment and it 
will produce the accurate result irrespective of any noise. The overall accuracy of the system is about 85%. 
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Figure 10. Accuracy matrix of different test case. 

This system is tested on four children who suffer from this disability. First of all we taught them in traditional 
classroom system based on text book and take a test and the score obtain by each individual are recorded. 
After few days we taught them the same content using our system and again arrange a test. Significant 
improvement was found between two score card. All the children secured better score than the previous one 
and the encouragement of learning among the children is remarkable. They find more interest and motivation 
to learn and remember things more quickly. Figure [11] shows some experimental result of our proposed 
system. 


Participating Children 

Score Secured 

Children A 

80 

Children B 

85 

Children C 

78 

Children D 

81 


Participating Children 

Score Secured 

Children A 

55 

Children B 

58 

Children C 

52 

Children D 

60 



Children A Children B Children C Children D 


[c] 

Figure 11. [a] Score obtain before after normal teaching [b] Score obtain after teaching by proposed prototype 
[c]Graphical representation of the experiment result. 
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6. LIMITATIONS AND FUTURE WORK 

In this proposed system some pre-defined word are stored in the vocabulary dictionary and only those word 
can be recognized by the Kinect sensor and corresponding image are drawn in the screen and mapped 
information will store in the database. These limitations can be overcome by making the vocabulary 
dictionary dynamic, therefore any word can be recognized by the system. More over this system can be used 
as a platform for device and appliance control at any place so that it will work as a assistive tool for aged and 
physically challenged people. 


7. CONCLUSION 

The main problem of the children with cognitive disability is the lack of interest and motivation to learn. 
Therefore a special type of care should be taken so that they can find themselves with higher self-confidence 
and stimulus environment for learning. The primary motive to achieve a mechanism for continues interest 
building towards learning has been significantly achieved. In this paper the proposed audio visual augmented 
interactive learning tool not only enhances the performance of the children effected by cognitive disability, 
but can also be used as significant support system towards elementary teaching to all level of children. The 
crucial accuracy of the system is 80-95% in achieving time versus learning outcome, which is a promising 
result and encourages the children to gain interest in continuous learning and make learning fun. This 
proposed first prototype is undergoing several real-life calibration like lighting condition, mobility, 
multilingual support etc. , refinement for individual proficiency and machine guided learning endeavored as 
a future goal of the work. 
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ABSTRACT 

The content of this paper will focus on both language and computer practices and how school age students develop their 
literacy skills in the two domains of ‘language’ and ‘computers’. The term literacy is a broad concept that has attracted 
many interpretations over the years. Some of the concepts raised by the literature apply to both language and computer 
literacy issues. Hence, this paper is intended to review the literature in areas: such as the definitional and conceptual 
issues of literacy; the development of language literacy (formally assessed in the areas of reading, vocabulary and 
comprehension) skills, but which also incorporates handwriting practices; computer literacy which applies to the 
acquisition and development of computer skills, particularly those associated with spelling and grammar checkers; the 
evolution of computer literacy to include new technological components emerge and interlock in computer usage to 
encompass Information and Communication Technology (ICT) literacy and its applications; in conjunction with the 
revelations of studies relating to computer use of spelling and grammar checkers and educational achievement. 


KEYWORDS 

Reading, Writing, Language and Computer Literacies, Spelling and Grammar Checkers. 


1. INTRODUCTION 

Today, “technological change happens so rapidly that the changes to literacy are shaped not only by 
technology but also by our ability to adapt and acquire the new literacies that emerge” (Leu et al 2004: 1591). 
In addition, Atkin (1998: 12) suggested that our current youth live in the emerging world, a world of 
transition and with a global and local focus: “life-long learners, learning to learn, contextualised and 
transformative”. The use of electronic medium/computers has influenced the way young people in what 
Tapscott (1998: 1) referred to as the Net-Generation, perceive and transform the traditional written language 
into a language of their own, shaped by short conventional text messaging and online-chatting and reading. 
Children (the Net-Generation) are more perceptive and willing to exploit the electronic devices by engaging 
in the digital culture to construct self-identity and community (Mountifield: 2006). These changes have made 
computer literacy skills more available to include interactions and communications through social events and 
practices (Street 2001: 11). The acquisition of language literacy differs from the acquisition of computer 
literacy in their achievement and learning development. The school aged children live in the emerging world 
(Atkin 1998), a world of transition and with a global and local focus. The Net-Generation has the opportunity 
to use the computer and their tools (spelling and grammar checkers) to present their work in a traditional 
form rather than in the variety that they invented. Many students trust the spelling and grammar checkers 
because they are very unsure of their own spelling and grammar (Sinclair 2010). 


2. RESEARCH METHODS 

I will adopt a triangulation approach including both quantitative and qualitative methods using a mixed 
methods design for two-phase study. In Phase 1 , language literacy tests consists of reading, comprehension 
and writing, will be administered to 150 year 9 students. Pre-survey was conducted to seek students’ 
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perceptions of the usefulness of the tools. The quantitative data from the first phase will sequentially be 
integrated into the second phase where qualitative methodology will be used to obtain a deeper understanding 
(McMillan and Schumaker 2006, Creswell and Garrett 2008) of the influences of the tools on the students’ 
English writing. To elicit qualitative data, I will observe the students in their classes, analyse their writing 
and interview them and their teachers. 


3. DEFINITIONAL AND CONCEPTUAL ISSUES 

Literacy definitions have expanded from an original focus on just reading and writing to include additional 
types relating to many aspects of contemporary society. Issues relating to literacy definitions have reflected 
many dimensions and explanations from different perspectives and disciplinary areas. Theorists such as 
Street (1984) have distinguished between an autonomous model and an ideological model of literacy. In the 
autonomous model, literacy is defined as a set of value-free skills, like decoding the printed words into 
sounds (decontextualising text) (Street 1995: 18 - 19). Viewed from this perspective - the acquisition of 
reading and writing skills is simply a cognitive process. The ideological model is recognized in a multiplicity 
of literacies and practices to specific cultural contexts. Both models of literacy have been interpreted in 
different ways by different scholars. For example, Blake and Blake’s (2005: 172) interpretation of the 
autonomous model is “the prevailing Western view of literacy, a single thought”. In extending a modified 
view of literacy into the social domain, Belisle (2006) included three complementary approaches to literacy 
that stood out in educational analysis: 

an autonomous model of literacy is based on the assumption that reading and writing are 
simply technical skills; a socio-culturcil model, based on the recognition of all literacies as 
socially and ideologically embedded; and a strong claim model based on anthropological 
statements of the revolutionary power of instrumented thinking processes (p. 52). 

The autonomous model has been criticised many times over the years as a result of questioning its 
strategies, applications and goal directions, particularly in response to the rapid development of technology 
and its wide use, by all ages, in contemporary society. 


4. DEFINING LANGUAGE AND COMPUTER LITERACIES 

Attempts to define language literacy have extended well beyond reading and writing. The varied disciplines 
defined their respective literacy in accordance with their research and study areas (functional, cultural, 
political and other literacies). An additional type of literacy is computer literacy. Hence, Street’s (1984) 
autonomous and ideological models of literacy that were subsequently replaced by the notions of literacy 
events and practices, partially apply to computer literacy. Many technological dimensions have been 
considered in moving to the current term of ICT and digital literacy (Avila and Moore 2012). 

Understandings of computer literacy includes literacy events with many dimensions underpinning literacy 
practices at the global level such as information, visual, technology and digital literacies (Cohen and Cowen 
2008). Taking into consideration the relation to computer literacy, Fehring (2010: 183) stated that there are 
two “interlocked components of the concept of multiple literacies”; one refers to the multiple forms of 
literacy now required to operate in the world of education and work; and the other concept of multiple 
literacies refers to the “multimodal and multidimensional aspects that the learning of literacy skills now 
encompasses”. Hence, Fehring (2010) asserted that the concept of multiple literacies has had a powerful 
influence on classroom practice: “Multimodal and multidimensional curricular have become the standard for 
students from the young age to lifelong learners” (p. 180). Ultimately, the young and the old have to become 
computer literate to effectively use ICT and its components such as the Internet and other digital resources. 
Despite the need to become computer literate, Blake and Blake (2005: 172) reported that the use of ICT also 
requires language literacy skills, “[ljeading and writing are [also] used [in order] to transmit information, to 
interpret, to respond to the expression of human thought.” Language literacy skills are fundamental 
acquisition for the exchange of information required in our society. 
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5. LANGUAGE LITERACY 

The social situations have changed and brought with them changes to the definition of language literacy with 
the additional emergence of new technologies in educational, domestic and workplace environments. Forster 
(2009: 12) reported that there is no single internationally accepted definition of ‘literacy’. However, the term has 
begun to address more complex understandings than when ‘being literate’ was defined by the ability to read and 
write. The definitions of literacy are of increasing breadth and reflect a growing emphasis on context. 

5.1 Reading Vocabulary and Comprehension as Parts of Literacy 

The major purpose of reading is the construction of meaning, comprehending and actively responding to 
what is read (Christie 2005: 1 - 4). Malatesha and Aaron (2010: 317) stated that vocabulary knowledge is a 
prerequisite and a critical factor in improving reading comprehension. Readers use their knowledge of text 
structures to build a coherent memory representation, and these structures or relationships are part of their 
cognitive representation (Luke (2012: 6). Narvaez (2002: 158) outlined the causes of individual differences 
in the comprehension of texts along two lines: “reading skill” and “reader’s knowledge”. 

Chen and Lee (2010: 127) discovered from their study that text rewrites using “social networking 
applications [such as the Internet using blogs, discussion boards and creating web pages] engage their 
students more effectively in interacting with the text”. New skills can be derived from using, for example 
‘Facebook’ (social networking site), according to Rowsell (2009: 108), “mediating identities through 
multiple modes and applications. . . . , shaping written text and visuals around diverse audiences that have 
shorter and longer timescales”. Moje (2010) acknowledged that these conflicting aspects, together with a 
probable lack of literacy skills in young people’s language development (due to lack of continued, sustained 
literacy instruction), create difficulties for many youth in secondary schools to read at even basic levels (p. 
50). She stated that reading at the secondary school level is more demanding and complex. Her reason was 
that “regular and explicit literacy instructions tend to diminish around Grade 6 (p. 49). 

Leino et al (2004: 252) included other studies in their article to assert that students who spend a lot of 
time reading on their own tend to be better readers than those who devote limited amounts of time to reading. 
From the autonomous view, Mckenna and Simkin (2008: 85 - 87) stated that “reading is a tool, or set of 
tools, for content acquisition”. Their research into technology applications that foster “reading growth have 
implications for content learning in digital environments.. . .The research into technology applications in 
reading can be divided into two categories: between “higher” and “lower” order processes”. They were 
described as “word recognition” and “comprehension”. 

5.2 Writing 

Writing and literacy stand in a complex relationship to one another. As children write in languages such as 
English, they look closely at how letters are used to form words and construct a system for the spelling 
(phonic) conventions of written text. As with reading, lots of writing in an alphabetic language improves 
phonic knowledge. Nevertheless, Boscolo and Mason (2001 ) concluded that writing can improve students’ 
learning by promoting active knowledge construction that requires them to be involved in transforming rather 
than only in a process of reproducing. Through writing, students have the opportunity to manipulate, 
integrate, and re-structure knowledge by using, and reflecting on their existing conception and beliefs in a 
continuous process of developing meaningful understanding (p. 85). 

5.2.1 Handwriting 

Handwriting has been largely forgotten in the literacy and ICT debates, but it is still needed to reinforce 
learning and language development. The temptations to ignore the development of handwriting skills are due 
to the advent of alternative modes of composing writing by using word processing and speech synthesis (De 
Souza and Towndrow 2010: 26). They stressed that handwriting is very important for students who still use 
pen and paper format in their exams. 
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6. COMPUTER LITERACY 

Computer literacy definitions vary depending not only on the different levels of users from regular users to 
power users (software developers, programmers and network infrastructure experts), but also on how literacy 
is perceived and applied by educational and industrial/workplace theorists. Computer literacy involves not 
only the understanding of what is possible with (and what influences the use of) computers, but also the 
physical use of combined equipment (peripherals) and software applications (Corbel and Gruba 2004: 23). At 
a less specialised level and from the autonomous view, computer literacy involves the knowledge of how to 
turn on a computer, start and stop software applications as well as save and print documents (Corbel and 
Gruba 2004: 24). In relation to software, Cohen and Cowen (2008: 546) defined computer literacy as “the 
ability to effectively use [autonomously] computer tools, [and software applications] such as word 
processors, spreadsheets, databases, [PowerPoint] presentation and graphic software”. From a possibly wider 
perspective, Moursund’s (2003: 9) definition of computer literacy, that also reflects an autonomous model, is 
“a functional level of knowledge and skills in using computers and computer-based multimedia as an aid to 
communication with oneself and others for the purposes of learning, knowing, and for using one’s 
knowledge ”. From an ideological view, computer literacy has evolved into a broad term that incorporates the 
use of the internet, ICT which drives language re-form/transformation and other digital devices. 

6.1 The Internet 

The Internet is an integral part of computer literacy. It is a “powerful tool and endless source of information, 
which is easy to find and easy to produce” (Knierzinger and Turcsanyi-Szabo 2001: 926). The effects of the 
Internet are deep and complex. A study by Leu et al (2007: 46 - 47) suggested that most of the adolescents 
who read the results page from a search engine [the Internet], . . . “do not actually read the items on the result 
page. Instead, the majority use a simplistic ‘click and look’ strategy”. More emphasis has been placed on new 
literacies as new technological devices appear, but also as their scope elaborates the ideological dimensions 
of multiple uses that emerge and interlock in expanded computer usage and its applications (Florian 2004: 8). 
The Internet has become more and more important for young people’s lives at school and at home. Ma et al 
(2008: 197) stated that “the Internet is affecting all subjects in K-12”. Ma et al’s study included those 
described by Tapscott’s (1998) term the ‘Net-generations’ who are fluent with digital technology, including 
all sorts of digital and electronic devices. The Internet use has also meant that “more children at school are 
practising cheating” (Ma et al (2008: 198). They added that there are “Web sites that provide free essays for 
students to plagiarize reports and term papers” (p. 199). They reviewed other studies which reported that the 
majority of their subjects (from different age group) responded that they copied and pasted from the Internet, 
“the characteristics of the Internet brought more convenience to digital plagiarism, particularly among middle 
school students” (p. 199). They used an example from one of the studies of a female student who forgot to do 
her homework. She went online and copied a paper in her handwriting believing the teacher would not find 
out what she did. Young people are more vulnerable to this kind of data capture since they are inclined to use 
the internet more interactively and purposefully. It is, therefore, necessary to ensure that ICT (Information 
and Communication Technology) is understood broadly. 

6.2 ICT Literacy 

ICT literacy is a broad term that includes multiple communication devices, various services and applications 
associated with it. ICT literacy is increasingly regarded as a broad set of generalisable and transferable 
knowledge, skills and understandings that relate to communication tools used to access, manage, integrate, 
evaluate and create information in order to function in a knowledgeable society (Ainley 2010: 2). ICT 
literacy covers the new and emergent technological devices combined, introducing new literacies (Internet, 
iPod and others) as they become available. Harris (2005: 34) stated that ICTs are “social information spaces”. 
They are designed as much for the reciprocal “sharing of information” as they are for “seeking and 
disseminating information”. He elaborates that “sharing” involves exchanging information amongst users and 
“seeking” implies going to sources outside one’s immediate social system. Out-of-school and in-school 
digital literacies are used by youth interactively and purposefully, in ways that are increasingly hypertextual, 
connected and communicative (Bussert-Webb and Diaz 2012: 5). 
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6.3 Language Re-form/Transformation 

The re-form of any language in the current era is often attributed to the speech-like characteristics of much 
electronic communication. For example, printed texts can often become ‘ad-speak’ and therefore 
unconventionally spelt and punctuated, for example, “btw” replacing by “by the way” (Tapscott 1998: 1 - 2, 
Christie 2005: 186). Often, people use digital tools to communicate, spell words using only abbreviations or 
one letter or number or a specific combination of letters, or letters and numbers to mean what the writer 
suggests. For example, “before” is replaced by “b4”, “later” is replaced by “18r”, "for" is replaced by "4" and 
“laughing out loud” is replaced by “lol” and many other re-formed words or expressions (Tapscott ( 1998: 1 - 
2); Christie 2005: 190) noted further opportunities for change and suggested alternative opportunities can be 
seen in multiple sound-spelling relationships. 

In line with Tapscott (1998), Mountifield (2006: 172 - 173) referred to the Net-Generation or Net-Gen. 
Such people were “born after 1982, [and] are the ‘digital natives’, always connected, highly mobile, able to 
multitask, format agnostic, comfortable in a visual-rich environment and able to move seamlessly between the 
physical and the virtual environments”. Despite their reputation as digital natives, according to Harris (2008: 
161), young people’s skills in effective navigation of today’s information landscape are actually somewhat 
limited. He elaborated that “young people are at a developmental disadvantage when it comes to evaluating 
digital media”. They are more likely to apply their own judgment based on the web sites rating and their choices 
are based on design and presentation features rather than the content. However, young people’s skills would 
progressively improve with “cognitive growth, education, and experience”. But Net-chatting and extended text 
reading in English are expected to have an even greater effect on writing (Naevdal 2007: 1113). 


7. COMPUTER TOOLS 

The computer software applications consist of many features (font size, text style, colour, WordArt and many 
others) and tools (spelling checker and grammar checker) for the users to enhance their works and improve 
their presentations. Galletta et al (2005) reported that the spelling and grammar checkers are common utilities 
found in many software packages. 

7.1 Spelling Checker 

The first tool to be described is the spelling checker. The user is prompted by the spelling checker to review 
individual words with incorrect spelling by instantaneously underlining the words with a squiggly red line 
(indicator of misspelt words) while they are writing (Hartley and Tynjala 2001: 165). When the user sees the 
prompts on the screen, he/she places the mouse pointer on the underlined word and right clicks to access the 
information. The words are compared to an electronic dictionary included in the software. I pre-surveyed one 
hundred and fifty Year 9 students on the usefulness of the spelling checker. The students’ responses are shown 
in Figure 1 . 

Stundents' perceptions of the spelling checker 



Very Useful Not useful I do not Left blank 

useful use it 


Figure 1. Students’ perceptions on the usefulness of the spelling checkers 
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Figure 1 shows that 90% of boys and 60% of girls perceived the spelling checker as very useful. Some 
students commented that “the electronic spelling checker enhances [their] English writing”. However, 
learning to spell enhances children’s English writing. Although, Sinclair (2010: 6) stated that there are at 
least two types of grammar definition: “one type is [describing] the way the language is used; and the second 
type emphasises the correct use and following a set of rules”. She stressed that when students use the spelling 
and grammar checkers and do not understand what the checkers are telling them, then they become confused 
which can affect their literacy skills development (p. 133). 

7.2 Grammar Checker 

Groups of words with grammatical errors are underlined with a squiggly green line as an indicator of 
sentence structure issues. A grammar checker is used to check for grammar, writing style, and sentence 
structure errors, but not all identified groups of words are necessarily ‘wrong’. A style feature such as passive 
voice may be underlined in a decontextualised attempt to make writing ‘simpler’. The tool can check 
documents for excessive use of a word or phrase, identify sentences that are too long, and find words that are 
used out of context such as ‘four example’ (Hong Wei and Davies: 2011). I also sought the students’ 
responses on the usefulness of the grammar checker. Their responses are shown in Figure 2. 



Figure 2. Students’ perceptions on the usefulness of the grammar checkers 


Figure 2 shows that 85% of the boys and 84% of the girls perceived the grammar checker from useful to 
very useful. Sinclair (2010: 6) added that many students trust the spelling and grammar checkers because 
they are very unsure of their own spelling and grammar. Consequently, they assume that the checker must be 
right. Sinclair (2010) included that the pitfalls are when the checker does not adequately recognise 
alternatives because the setting is inappropriate, for example, using a US dictionary for work in the UK, 
resulting in spellings such as ‘behavior’ rather than ‘behaviour’; sometimes the checker can’t distinguish 
between frequently confused words which can often make students change a correct word to a wrong one (p. 
140). Since the advent of word processing and spell checkers, according to Moats (2006: 1 - 2), some 
educators have argued that spelling instruction is unnecessary. 


8. STUDIES EVALUATING SPELLING AND GRAMMAR CHECKERS 

Several studies have investigated and evaluated the use of the spelling and grammar checkers and their 
implications for language literacy development for school-age children and their English writing. Facilitated 
by the electronic medium, according to Lam (2008: 1209) “ ... the English Language is becoming 
increasingly tied to the cultural expression of various groups of native and non -native around the world”. 
Conversely, in other language research, Heift and Rimrott’s (2008: 196) study revealed that “while the 
number of correct responses was significantly higher when the system provided a correction list, there was 
also significantly less learner uptake for the feedback type that did not provide any correction suggestions”. 
Moreover, students were far more successful in submitting the target word if it appeared in the spelling 
checker suggestion list. In contrast, if the target word was absent from the suggestion list, students picked a 


168 




12th International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2015) 


wrong replacement word from the list. Heift and Rimrott’s (2008) study concluded that the order in which the 
words appear in the suggestion list seemed to be an influencing factor for students to select one word over 
another. In other words, their participants picked the word that appeared in first position in the spell checker 
list more than in any subsequent position. 

Figueredo and Varnhagen (2006) conducted their study in Canada titled ‘Spelling and grammar checkers: 
are they intrusive?’ Their subjects were twenty- five freshmen, 20 English majors and 20 graduate students 
revised two essays on a computer, one with the spelling and grammar checkers, and the other with a dictionary 
(p. 723). For the checker condition, the spelling and grammar checkers were turned on, and the spelling and 
punctuation errors were flagged in the text by red and green squiggly lines respectively. They modified the 
correction options offered by the checkers. For the spelling errors, they created a word list of potential 
suggestions for spelling corrections. If the participant clicked on a flagged error, a pull down menu would 
appear with three potential choices. The correct choice and two foils were placed in different positions in the 
pull-down menu. For example, ‘retrospect’, Tetrespect’, ‘retrispect’ were supplied for the error ‘retraspect’ (p. 
726). They used the same method for punctuation errors. The participants completed two essays. They revised 
the first essay on a word processor and had access to a dictionary for the second essay. Their findings revealed 
that content revisions were not significantly different across technology conditions, but all student groups were 
able to correct more surface errors with the aid of the checkers than they were with the dictionary. They also 
found a correlation between spelling ability and correction in the dictionary condition but not in the checker 
condition. They concluded that “the main source of difficulty for students, at that level, in the dictionary 
condition was in detecting the surface errors without the help of the checkers” (p. 730). 

Using a special program, ‘Grammatik V’, Hong Wei and Davies (201 1 ) sought to assess the effectiveness 
of a popular grammar and style checker. They conducted their study in Thames Valley University and 
Richmond College in London. Seventeen students of English as Foreign Language, who came from six 
different subject disciplines were involved. During the trial, the students were required to produce two 
versions of each writing sample. One sample was produced prior to using the program. The second contained 
the changes made during the trial. Both samples were collected in order to provide information on the effects 
of using the program (p. 2). Their findings revealed both mechanical problems and spelling. The mechanical 
problems were to do with ‘capitalisation’, 'punctuation’, ‘infinitive forms’, ‘number styles’ and others. The 
spelling errors were separated from the others in order to highlight specific issues and to avoid misleading 
conclusions (p. 3). They reported that the program proved much less successful in identifying and relating 
subjects with verbs when they used the modifiers in between or when they appeared in complex sentences. 
They concluded that the “effectiveness of using the current [Grammatik V] program to check students’ 
writing for academic purposes, on the whole, is not very satisfactory, although the different ratios of correctly 
and misdetected and questionable problems did suggest that there were variations of performance in the three 
areas the program focused on: mechanics, grammar and style” (p. 10). 

Some researchers have investigated just how much people rely on the little squiggly red and green lines in 
Microsoft documents. They have found that, while attention to what is identified by the lines improves the 
quality of poor writers, it makes good writers worse - since they start to rely on the spelling and grammar 
checkers completely, while ignoring their own instincts (Olsen and Williams 2004: 1020 - 1022). They 
found that “. . . spell checkers do little to deal with issues such as the use of homonyms, such as the word 
‘desert’ versus ‘dessert’. It will let you eat ‘desert’ as well as die from thirst in the ‘dessert’” (p. 1021). 
Grammar checkers work from a set of rules about when a plural noun is used with a singular verb in typical 
cases, for example, “is” versus “are” usage, but they also fail to misdiagnose many cases as well (p. 1022). In 
line with Olsen and Williams 2005, Galletta et al (2005: 82) investigated different versions of word 
processors content-related features (spelling and grammar checkers) which they called “language-checking 
software”. They found false positives and negatives: “the false negatives, where the language-checking 
software fails to detect true errors”. The example they used in their study was “[g]o ahead with the complete 
role-out” where “role” was not flagged to be replaced by “roll”; and false positives, where the software 
detects problems that are not errors”. They illustrated another example for the sentence “Multiple regression 
was run”. The spell checker underlined “regression” and suggested it to be changed to “regressions”. If and 
when the user follows that advice, other difficulties cascade down the errant path. The word “was” was then 
underlined and the suggestion was made to change the word to “were”. Following the suggested advice, 
distorted the true meaning of the original sentence (p. 3). 

Another study conducted by Fandrych (2001: 1 - 12) investigated the spelling and grammar checkers 
assistance, in two software applications (Microsoft Word and WordPerfect), and the consequences for second 
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language learning and teaching. She selected case studies of English in South Africa. One of the students 
typed ‘inorder’. It was flagged to be replaced with “one of the following: ‘ignored’, ‘ironed’, ‘ironware’, 
‘intruder’, ‘inured’ - all of which would be syntactically inappropriate” (p. 4). More problems were detected 
that contained misleading advice to restructure the sentence using the grammar checker. She concluded that 
“word processors cannot solve all the problems many users face when composing texts, especially if English 
is not their first language” (p. 12). As a consequence, according to Vascellaro (2006: 1 - 3), immersion in 
new technologies improved student attitudes and behaviours but had little overall impact on student 
achievement. At a higher academic level, a study was conducted at the University of Pittsburgh, 
Pennsylvania, United States of America, involving 33 undergraduate students to look at the differences in 
reader’s ability to correct spelling and grammar errors with and without the assistance of spelling and 
grammar checking tools. The participants were given a letter to correct. Half of the group utilised spelling 
and grammar checkers, and the other half did not. Using Skills Assessment Test (SAT) which consists of 
three sections: critical reading, mathematics and writing. The results were “those with strong verbal skills 
rely heavily on the grammar and spelling features of word processing software, thereby hurting the accuracy 
of their documents”. The group that did not use the software performed as expected in relation to each 
student’s verbal score. She concluded that “[ujnfortunately, a computer is much better at numbers than it is at 
doing grammar and spell-checks” (Osborne 2003: 1 - 8). From the above studies, it is clear that the spelling 
and grammar checkers are not helping the digital natives to improve on their traditional literacy development. 


9. CONCLUSION 

Language literacy acquisition differs from computer literacy, and the two literacies require different skills 
from each other as shown by their definitions in Sections 3, 4, 5 and 6. Making connections between the two 
literacies leads to further influences on and implications for the development of traditional language learning 
as shown in the literature. The literature suggests that being competent in literacy implies that one knows 
which practices, attitudes and values are appropriate in a given situation. While education is often focused on 
mastering the autonomous dimensions of a given literacy, the ideological dimensions can present much more 
compelling and in depth challenges. The prominent messages stemming from the literature are that young 
children should be developing an enriched vocabulary as an indicator of oral language proficiency which is 
essential for comprehension of both oral and written language. However, technological changes have 
happened so rapidly that changes to literacy are shaped not only by technology, but by our ability to adapt 
and acquire the new literacies that emerge with its applications (Leu et al 2004; Florian 2004). Good 
examples are shown in Figures 1 and 2, the students perceived the spelling and grammar checkers as very 
useful in enhancing their English writing. 

The conducted studies evaluating relationships between computer use and educational achievement, 
Olsen and Williams (2005); Hong Wei and Davies (2011), suggested that the mere expanded use of 
computers is not a guarantee for the acquisition of proficient language literacy. Their conclusions are 
complex. The students are not developing their language literacy skills sufficiently, or practising their 
speaking and writing skills, by engaging in direct interactions with the electronic medium. These skills can be 
developed by reading, expanding their vocabulary and comprehension and by presenting their own written 
work (Ma et al 2008). 

Other studies evaluated the spelling and grammar checkers and just how much school-age children rely 
on the little squiggly red and green lines in word processing documents. Their findings indicated that while it 
improves the quality of poor writers, it makes good writers worse by enabling or encouraging them to ignore 
their own instincts (Olsen and Williams 2004). Word processors cannot solve all the problems many users 
face when composing texts. There were many issues falsely advised by the tools (Fandrych 2001). In 
examination of different versions of word processors, Galletta et al (2005) and Fandrych (2001) have shown 
similar results that the software failed to detect true errors. If and when the user follows the advice suggested 
by the programs, other difficulties cascade down the errant path. 

There are implications in the use of spelling and grammar checkers as the literature revealed. Therefore, 
teachers should teach students how to use the electronic spelling and grammar checkers and encourage them 
to use the dictionaries as well. So the students can use the correct spelling rather than guess and select the 
words and sentences randomly from the provided list by the tools. 
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ABSTRACT 

Scaffolding is crucial as transfer of learning does not occur naturally and teaching-learning strategies found to be 
effective for experts may not be suitably adopted as is for novice learners. Furthermore, opportunities are often “found” 
or “made.” The quality of solutions, however, is mediated by different conceptualizations of contexts. We aim to 
investigate whether design thinking and metacognitive reflective scaffolds can help graphic design students to design on a 
different canvas, i.e., for industrial design, in order to increase their job opportunities. Findings indicate that students are 
able to identify and match patterns, theorize based on their graphic design knowledge (to transform fuzzy association 
patterns into fuzzy associative rules) and analyse existing designs in the market to create their own designs. The 
significance of the study are useful to the learning of Science, Technology, Engineering and Mathematics, co-design of 
designs/uses for smart technologies, industrial design, and embedded cognitive systems, such as robotics and Internet of 
Things (IoT). 


KEYWORDS 

Conceptualization; design thinking; metacognitive reflective scaffolding; creative thinking; graphic design-industrial 
design; acquisition of expertise; transfer of learning 


1. INTRODUCTION 

Economies are increasingly knowledge-based, i.e., driven by knowledge, information and technology. Hence, 
the development of skills throughout life becomes critical. A McKinsey report on Education to Employment, 
highlights two critical issues in the 21 st century: high youth unemployment rates and shortage of workforce 
with critical job skills, due to a mismatch in skills. For example, in 2013, 73.4 million young people are 
unemployed. In addition, 58% of employers surveyed think that entry-level new hires do not have the 
necessary skills. Consequently, in a dynamically changing and interconnected world, how can we best equip 
our students to cope with continuous changes, and to learn continuously? 

UNESCO (2015) thus highlights the need for educational innovations, aimed at: 

1. developing educational and training contents, pedagogies and knowledge to improve 21st century 
skills; 

2. inculcating significant changes in values, mindsets, practices, behaviours and skills for now and in the 
future; and 

3. promoting and translating principles of lifelong learning in skills development. 

These aims focus not only on effective re-skilling and adjustment to skills mismatch (first objective), but 
also to the development of dispositions which would contribute to productive and sustainable development 
(second and third objectives). Subsequently, three sets of skills have been identified: (i) foundation skills, (ii) 
specialized skills and (iii) transversal/transfer skills. UNESCO further points out that these three sets of skills 
are equally important if we want to enhance the potential of education and the human innovative spirit to 
address authentic pressing issues facing our world today. These skills are also supported by Outcome-based 
Education (OBE). 
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1.1 Objectives 

With the above scenario as background to our study, we investigate whether design thinking can help graphic 
design students to more effectively build knowledge, develop creativity and subsequently, design on a 
different canvas, i.e., for industrial design. We choose industrial design because of the similarity between the 
two disciplines. Industrial design mainly focuses on function and form, and is concerned with aesthetics, 
psychology, market analysis, user interface design and information design, ultimately, leading to interaction 
design (Pulos, 1988). Given our sample students’ knowledge and experience with packaging design, we are 
interested to investigate whether this knowledge can be transferred to industrial design. 

1.2 Hypotheses 

Studies by Papert (1980) and many other learning sciences research, attest that computers are construction 
tools, with great capacity in expanding what people can create and what they can learn in the process. Their 
constructionist research provide evidential indications that many of our best learning experiences are when 
we are engaged in designing and creating things, particularly things which are meaningful either to us or to 
others around us. Bereiter (1995) contends that learning dispositions and beliefs are key in determining 
learning outcomes. 

Subsequently, based on their arguments, we contend that knowledge construction and consequently, 
learning outcomes (OBE, 1994) and transfer, are built on students' dispositions. We have scaffolded the 
development of such dispositions through design thinking, framing and metacognitive reflection in our 
studies (Lee & Wong, 2013, Lee & Wong, 2014a, Lee & Wong, 2014b). Though the first study involves 
low-fidelity prototyping and the subsequent ones high fidelity prototyping, findings are consistently positive. 
This leads us to our first hypothesis. 

Hypothesis 1: If students fully practice design thinking, and apply our metacognitive reflective scaffolds, 
they may be able to better link between theory and practice. 

This study is different from our prior studies due to the domain concerned, i.e., graphic design. In this 
study, we incorporate Art and Design theories as methodology. We regard Communication theory as primary 
in design. Initially perceived as a linear process (Shannon & Weaver, 1949), Schramm’s (1971) 
Communication theory involves sending messages and correspondingly providing and interpreting feedback. 
User experience is important in determining the quality or success of communication. The emphasis on fields 
of experience between the sender and receiver implies that life experiences enhance the quality of 
communication when they overlap. This leads us to our second hypothesis. 

Hypothesis 2: If students are able to create user experiences based on Communication theory and related 
theories, they should be able to design more creative industrial designs. 

Our research model is illustrated in Figure 1. 
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Context: Society/Culture 


Scaffolds: Framing 



Figure 1. Research model for this study 


1.3 Significance of the Study 

First, Verganti (2008) argues that innovation is often through increasing the emotional and symbolic value of 
products. However, the underlying mechanisms are still poorly understood. Second, we hope to increase 
cognitive access, i.e., equal opportunity to understand and learn, regardless of ability, race, ethnicity, 
socioeconomic status, and gender as espoused by Outcome-Based Education (OBE, 1994) by developing 
epistemic agency (Scardamalia & Bereiter, 1995) through our design scaffolds Third, we argue that smart 
communities encompass not only the use of smart technologies but also the co-design of designs/uses for 
these smart technologies. In line with natural computation, this implies a symbiotic relationship between the 
built environment and its ecosystems. A greater awareness of this symbiosis may be useful industries such as 
industrial design, and embedded cognitive systems, such as robotics and Internet of Things (IoT), due to the 
translation of epistemic agency to autonomicity in software agents. 


2. RELATED WORK 

Consistent with the nature of problem-solving in authentic environments, an experimental game-thinking 
approach which captures the “fuzzy” characteristic of ideation/ innovation contextualized within the 
ecosystem(s) is adopted throughout our study. The following sections explain why. 

2.1 Socio- Cognitive Framework and Framing 

One of the approaches directly related to the theory of constructivist learning, experiential learning (Papert, 
1980; Bereiter, 1995) espouses that learning is constructed as a response to each individual’s experiences and 
prior knowledge (sense-making); i.e., that learning occurs through active exploration (learning-by-doing) 
within a social context. 

Following from this, we contextualize the design challenge within Engle’s (2006) sociological 
framework. Her framework espouses the constructivist-constructionist views, where technology and learning 
communities are key mediators which improve sense-making and user experience between interconnected 
contexts. Given that design is a communicative process, the sociological framework provides a more holistic 
context within which learning can be framed. 
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We choose this framework because it provides a situative theory of transfer whereby interconnected 
contexts create a coherent framework within which students can generatively use what they have learnt. 
Thus, the ability to frame accurately is crucial to epistemic agency as it provides the reason/ motivation/ 
epistemology for learning. In other words, if students comprehend and are able to relate to their personal 
goals, they are more likely to use the content-based supports more effectively. Hence, framing within a 
socio-cognitive framework will enable instructors and researchers to develop more comprehensive 
explanations underlying mechanisms of transfer. Transfer is scaffolded/facilitated because as Neumier (2013) 
points out, the external perspective affords the development of a metaposition and the development of 
metaheuristics. 

2.2 Design Thinking 

Design thinking is interdisciplinary and focuses on context, empathy and user experience (Brown & Wyatt, 
2010). Grounded on inquiry-based learning, Dunne & Martin (2006) recommend design thinking because 
problems are framed, questions are asked, ideas are generated, and answers are obtained. These are carried 
out iteratively, Figure 2 illustrates the design thinking processes. More importantly, being user-centred, 
outcomes are emergent. As such, the design thinking process allows not only information and ideas to be 
organized and choices to be made typical of problem-solving, but also insights to be gleaned, situations to be 
improved, and non-prescriptive/didactic knowledge to be developed. 

We choose design thinking because it best captures the game-like experimental spirit and balancing of 
multiple decision criteria inherent in game design and game -based learning. Furthermore, our prior studies 
(Lee & Wong, 2013; Lee & Wong, 2014a, Lee & Wong, 2014b) with inquiry learning (first study) and with 
design thinking (second and third studies) are successful in building knowledge and in developing creativity. 


Learning about the brainstorming and Returning to your original 

audience for whom you coming up with creative user group and testing 

are designing solutions. your Ideas for feedback. 



Redefining and focusing your Building a representation of 

question based on your insights one or more of your Ideas to 

from the empathy stage. show to others 


Figure 2. Design thinking phases 


3. METHODOLOGY 

3.1 Sample 

Sampling is purposive. Our sample students are 16 graphic design undergraduate students in a private 
university in Malaysia undertaking the course Design Reflective Practice, a consequent course after Design 
Research. All of them are final year students. They are the population for the course. In both courses, design 
thinking is taught and applied as an addition to the syllabus; serving as the binding thread across course 
content. The students’ backgrounds are mixed, from both Arts and Science (Engineering). Experimental 
duration is 13 weeks. 

3.2 Procedure 

Similar to our prior work, we design based on Marini and Genereux’s (1995) context, task and assessment. 
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Context: We frame learning of research skills as increasing the possibilities of job opportunities. First, we 
ask students what they think industrial training would be like, employers’ expectations of them, their 
designs/products and self-reflection on their own strengths and weaknesses. These form the primary means of 
framing. A secondary means of framing is through the design of the assessment. Subsequently, students are 
presented with multiple lectures on various Art and Design movements and theories and have experimented 
with these ideas as in-class projects. 

Task: Thus far, their in-class tasks have been to produce digital media Art to communicate. For the two 
hour lesson near the end of the course, the objective is to gauge how much they have learnt from the earlier 
lessons. It functions as one of the embedded assessments in the course. Students are asked to discuss what 
and how they would design a wallet for their loved ones and why. This is based on an example on design 
thinking by Stanford University’s d.school. Subsequently, they are given the task of designing a wallet for the 
high-end market (a different target market from the example) where money is not a concern. They are 
allowed to incorporate technology if they deem suitable. Feasibility is not the primary concern as technology 
can be developed if the idea is marketable. 

Assessment: Their end-of-course final assignment is based on the use of theories. The assessment criteria 
are similar throughout their reflective practice exercises and assignments. In this study, we regard fun and 
interactivity as contributing to sustainable user experience. As such, students are evaluated mainly based on 
fun, interactivity and sustainability. The assessment rubric used was refined from our prior studies (Lee & 
Wong, 2013; Lee & Wong, 2014a, Lee & Wong, 2014b). It is as shown in Table 1. 

Table 1. Refined assessment rubric 



A 

B 

C 

D 

Fun 

Users are addicted/ 

intrinsically 

motivated 

Users find it fun 
and don’t mind 
spending 2-3 hours 
per day at the 
Website/ 
application. 

Users find it 
somewhat fun but 
wouldn’t want to go 
to the Website or 
use the application 
unless they have to. 

Users find it boring 
and dread to go to 
the Website or use 
the application. 

Interactivity 

Users are actively 
creating and/or co- 
creating objects and 
interactions which 
will result in higher 
quality objects and 
more meaningful 
interaction. 

Users are creating 
and/or co-creating 
objects and 
interactions and the 
outcome are more 
objects and more 
interactions which 
are beautiful to 
them. 

Users are creating 
and/or co-creating 
objects and 
interactions. 

Users are not 
creating and/or co- 
creating objects and 
interactions. 

Sustainability 

Three to four 
dimensions of 
application. 

Two to three 
dimensions of 
application. 

One dimension of 
application. 

Failure to sustain. 

Problem-solving 

Problem and 
hypotheses properly 
formulated across 2-3 
domains. 

Problem and 
hypotheses properly 
formulated across 2 
domains. 

Problem and 
hypotheses properly 
formulated across 1 
domains. 

Problem and 
hypotheses 
improperly 
formulated. 

Significance/ 

Outcome can actually 

Outcome needs to 

Outcome needs 

Outcome is purely 

contribution 

be used in society 
directly. 

be adapted slightly 
before it can be 
used in society. 

major revision but 
can be adapted 
before it can be 
used in society. 

adoption of an 
existing solution. 

Novelty 

The outcome is 
original in relation to 
the existing work 
locally and 
internationally. 

The outcome is 
original in relation 
to the existing work 
locally. 

The outcome is 
original in relation 
to the existing work 
in the local 
community. 

The outcome is not 
original. 
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4. FINDINGS 

All students incorporate technology and whatever they perceive as being hot and uptrend in the market in 
order to optimize user experience and perception of classiness leading towards perception of the brand. All 
designs indicate top concerns with regards to security followed by customization to individual style. The 
differences in their designs lie in their emphases on what they perceive as the customers’ needs, preferences 
and desired user experience. In addition. Group l’s design is more inventive and multi-dimensionally 
functional whereas Group 2’s design is functionally adaptive and Group 3’s design focuses more on style 
than functions. Examples of their designs are presented below: 

Group 1: Sensor/face detection-unlock function, touch-screen ordering function (the chosen button will 
eject the card which they want to use so users do not have to rummage through their wallets), charging by 
shaking (movement detection) so no need to worry about charging anywhere anytime (even while walking), 
colour change, water proof, light-weight and GPS in case lost. 

Group 2: Finger/palm print recognition, the wallet is digitalized totally, functionally similar to a laptop in 
the sense of both sides of the wallet being able to swivel on different planes in any desired position and the 
screen being electronic so that all necessary items (assuming that these items are also digital as is possible in 
the future including ID and cash), magnetic open/close flap, USB port at the bottom left, and a cover which 
can change based on nanotechnology, GPS in case lost. 

Group 3: Thumb print recognition in order to unlock, changeable illustration with black background for 
day and white background for night. 

Each group has targeted only one perceived market. However, considering the beauty and desirability, of 
each design outcome, we contend that the diversity in design outcomes is able to reach wider and more 
diverse markets for an organization. 


5. CONCLUSION 

The discovery of knowledge is usually fuzzy and hence, it makes sense to base knowledge discovery on what 
is known, i.e., on pattern recognition and subsequently, hypothesize, experiment and consequently, formulate 
new patterns/ principles/ generalizations (defuzzification). 

Fuzziness during knowledge discovery is natural and even encouraged due to the advantages and 
possibilities of emergent outcomes from such experiments. We hope that based on our design factors 
(constructionism, design thinking and metacognitive reflective scaffolding as embedded assessment) and 
findings, cognitive access will improve despite our limited cognitive processing capabilities. Subsequently, 
with the increase in application of eclectic interdisciplinary student-centred approaches, we hope that 
students will be able to become better conceptualizers and designers; able to develop and sustain the ability 
to apply and synthesize knowledge learnt to create practical, useful new knowledge through interactions with 
the learners' local and international communities and that these designs will become more effective, efficient 
and most importantly, fun, generative and, meaningful. 

The next question is can we accept and find new ways to encourage more meaningful experimentations in 
the Sciences, where logic is more dominant and knowledge and representations are more well-structured? 
How can we make learning more fun, interactive and sustainable in the Sciences and increase the number of 
students taking up Science, Technology, Engineering, and Mathematics (STEM) and its variants in its 
applications to different domains e.g. e-commerce, social entrepreneurship, healthcare? 

Our sample size is small and there may be extraneous variables. At this moment, we can only claim that 
design thinking and our metacognitive reflective scaffolds help students to be more aware of salient design 
factors that they need to consider to reach the perceived market and subsequently, to improve lifelong 
learning, i.e., epistemic agency resulting in meaningful and sustainable knowledge acquisition and 
knowledge building. 
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Future work involves broadening our framework to include wider considerations. We intend to include 
lessons from these two studies as well as studies from the creative industries towards enhancing STEM 
education and transfer of learning to meet the challenges of the 21 st century. We will also include framing 
from an entrepreneurial perspective and an adapted LBD-based reflection grounded on our findings, to 
develop deeper understanding of design and design thinking towards more meaningful and sustainable social 
innovations. 
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ABSTRACT 

Advancements in technology and liberalization of educational opportunities have transformed the knowledge landscape 
into an emergent innovation incubator, where sense -making and creativity in and for ill-structured environments are 
especially relevant. In addition, opportunities are often “found” or “made.” The quality of solutions, however, is mediated 
by different conceptualizations of contexts. As such, we need new teaching-learning models which help students to 
become more effective and creative knowledge builders and creative innovators. In this paper, we present a summary of 
our prior work towards this objective, our framework and the refined assessment rubric derived from our various studies. 
Our design factors/scaffolds are: design thinking as design methodology, metacognitive reflective scaffolds and 
formative assessment as framing technique. We hope our findings can be applied to the learning of Science, Technology, 
Engineering and Mathematics (STEM) and its variants. Future work may involve its application to MOOCs or Cloud 
Computing. 
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1. MOTIVATION 

Crucial to the discovery, creation and sustainability of competitive advantage, knowledge diversity has 
resulted in and encouraged interdisciplinary methodologies. These encompass business, technology and 
human factors. As such, educators are challenged in terms of teaching-learning beliefs and practices with 
regards to how they should best equip/prepare students for the workplace. This is true especially in Multi 
Online Open Coursewares (MOOCs), where students are expected to be independent learners and also where 
dropout rate is sometimes high. 

Rentroia-Bonito, Jorge and Ghaoui (2009) point out that in order to address complex and 
multidisciplinary challenges, development teams need proper design techniques to build effective learning 
experiences. Thus far, they find that the literature does not indicate solid quantitative approaches which 
support learning-centered design where student needs and their immediate and broader contexts are 
considered. Hence, their study explores motivation to e-learn as a variable to designing technology-supported 
learning experiences. They aim to identify what motivation-related variables are critical for student 
engagement in learning online as the basis for a specific, bottom-up and quantitative design technique. Their 
study on the importance of a set of motivation -to-e-learn variables in a real instructional setting has led to an 
exploratory two-factor structure explaining 96% of motivation to e-learn. 

This two- factor structure (presented in Table 1) contributes towards quantitatively understanding and 
cost-effectively improving the link among learning design process, supporting systems and students in terms 
of continuous assessment of what students value most systemically. Courseware accounts for 81% of 
variance whereas organizational communication, accounts for 16% of the common variance. More 
importantly courseware factors explains four times more variance than organizational communication, 
indicating that it has a higher influence on motivation to e-learn and thus deserves more focus and priority 
due to its potential impact. Organizational communication however, influences readiness to e-learn. Among 
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these, the three factors with the highest mean are instructor support, accessibility to contents and 
easy-to-use-interface. We also note that these salient factors include beliefs (the affective domain), indicating 
that any systemic solution has to involve not only courseware and organizational communication, but also 
students’ beliefs/dispositions. This finding is consistent with Perkins and Salomon’s (1992), Bereiter’s 
(1995) and Engle’s (2006) propositions on transfer of learning. 


Table 1. Rentroia-Bonito, Jorge and Ghaoui’s (2009) two-factor structure and factors/variables which load on them. 


Courseware 

Organizational communication. 

Accessibility to contents 

Resource availability 

Security and data protection 

Instructor support 

Personalized feedback 

Institutional support 

Easy-to-use interface 

Feeling part of learning group 

Flexibility in content presentation 

Like studying subject matter 

Aesthetic content presentation 

Belief: E-learning contributes to my learning objectives 

Belief: I can learn this subject online. 

Belief: Adequate communication with instructors 


We are concerned with the design of embedded formative assessment as personalized feedback (part of 
instructor support, accessibility to contents) due to the above scenario and because of our concern with 
cognitive access. Cognitive access extends beyond digital access to include comprehension, and 
interpretation. Interpretation varies depending on the individual's prior learning and culture. As cognitive 
access (sense-making) becomes more critical, we need to provide structure and scaffolds to help students to 
develop their own learning initiative and learning path. We next discuss our suggestions towards this end. 


2. LESSONS LEARNT 

Based on the learning sciences’ literature, we learn that: 

a) transfer of learning is important because if students are able to transfer their learning to different 
contexts, outcomes may be the discovery or creation of opportunities; 

b) promotion of transfer of learning based on modelling (e.g. Kolodner’s (1995) Learning-by-Design, 
Engle’s (2006) socio-cognitive framework) and visualization (e.g. Bhatta, S. and Goel (1997); Ashok’s 
(2011) study on modelling of the acquarium ecosystem) is encouraged. This is because transfer can occur 
at different levels from the elemental to contextual; 

c) goal-seeking analyses and what-if analyses can lead to drill-down analyses. However, it is the 
context, which gives meaning to these analyses. 

To develop design thinking skills and a disposition for social innovations, we have used metacognitive 
reflective scaffolds grounded in inquiry-based formative assessment. 

2.1 Design Thinking as Design Methodology 

Brown and Wyatt (2010) observe that conventional research can point toward incremental improvements. 
However, these do not usually lead to significant breakthroughs. As such, they point out that conventionally, 
designers have emphasized on enhancing the look and functionality of products. However, recently, they 
have begun to utilize design tools in order to address more complex problems. This is because systemic 
problems require systemic solutions. The change in dynamics necessitates broadening approaches in order to 
create entire systems to deliver products and services. For example, social challenges require systemic 
solutions that are grounded not only in the client’s or customer’s needs, but also form and function, as well as 
distribution channels. Without considering the whole system, solutions may not be sustainable. 

As mentioned earlier, design thinking is inherently optimistic, constructive, and experiential. The 
beliefs/epistemology underlying design thinking is illustrated in Figure 1 . 
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Show Don't Tell 

Communicate your vision in an impactful and 
meaningful way by creating experiences, using 
illustrative visuals, and telling good stories. 



Focus on Human Values 

Empathy for the people you are 
designing for and feedback from these 
users is fundamental to good design. 


Craft Clarity 

Produce a coherent vision out of messy 
problems. Frame it in a way to inspire 
others and to fuel ideation. 


Embrace Experimentation 

Prototyping is not simply a way to validate your 
idea; it is an integral part of your innovation 
process. We build to think and learn. 



Be Mindful Op Process 

Know where you are in the design process, 
what methods to use in that stage, and 
what your goals are. 


Bias Toward Action 

Design thinking is a misnomer; it is more about 
doing that thinking. Bias toward doing and 
making over thinking and meeting. 



Radical Collaboration 

Bring together innovators with varied 
backgrounds and viewpoints. Enable 
breakthrough insights and solutions to 
emerge from the diversity. 


D.MINDSETS 


Figure I . Design thinking mindsets 

Source: Stanford’s d. school’s bootcamp bootleg 

Being a Human-Centered Design methodology, design thinking naturally incorporates consumer insights 
as the first design space. In 2001, IDEO transformed from designing consumer products to designing 
consumer experiences. With this change, design thinking develops our ability to a)be intuitive; b)recognize 
patterns; c) construct ideas that have emotional and functional meaning; d) express ourselves in media other 
than words or symbols. In short, IDEO positions design thinking as the integration of feeling, intuition, and 
inspiration with the rational and the analytical. The consequent methodology is thus a system of overlapping 
spaces rather than a sequence of orderly steps. Three main spaces are most well-known: inspiration, ideation, 
and implementation. 

Inspiration emphasizes on empathy through immersive experiences and understanding. These are 
subsequently validated through in-depth and rapid prototyping. Prototyping (transforming ideas into actual 
products and services, which are then tested, iterated, and refined) is crucial as it not only validates, but also 
enables the discovery of unforeseen implementation challenges and unintended consequences; and assesses 
the feedback and viability of new market approaches critical to more sustainable long-term success. 

2.2 Formative Assessment as Framing and Reflective Technique 

Mislevy, Steinberg, Almond and Lukas (2006) point out that effective assessments are key in enabling 
teachers to gather information about students’ knowledge and the depth of their understanding of subject 
matter. In fact, their studies indicate that assessment experts suggest assessments be designed grounded in 
evidentiary reasoning, i.e., the assessment activity is specifically designed to elicit evidence about the student 
knowledge that is of interest. 
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Based on the above literature, in our prior studies (Lee & Wong, 2013, Lee & Wong, 2014a; Lee & 
Wong, 2014b), we embedded metacognitive reflective scaffolds (Schon, 1996) as formative assessments, 
integrated with design thinking within progressive inquiry/problem-based learning. Tangible prototypes are 
produced for the latter two studies consistent with design thinking. Our assessment rubric refined throughout 
our prior work is presented in Table 2 below. 


Table 2. Refined assessment rubric. 



A 

B 

C 

D 

Fun 

Users are addicted/ 

intrinsically 

motivated 

Users find it fun 
and don’t mind 
spending 2-3 hours 
per day at the 
Website/ 
application. 

Users find it 
somewhat fun but 
wouldn’t want to 
go to the Website 
or use the 
application unless 
they have to. 

Users find it 
boring and dread 
to go to the 
Website or use the 
application. 

Interactivity 

Users are actively 
creating and/or co- 
creating objects and 
interactions which 
will result in higher 
quality objects and 
more meaningful 
interaction. 

Users are creating 
and/or co-creating 
objects and 
interactions and 
the outcome are 
more objects and 
more interactions 
which are 
beautiful to them. 

Users are creating 
and/or co-creating 
objects and 
interactions. 

Users are not 
creating and/or co- 
creating objects 
and interactions. 

Sustainability 

Three to four 
dimensions of 
application. 

Two to three 
dimensions of 
application. 

One dimension of 
application. 

Failure to sustain. 

Problem-solving 

Problem and 
hypotheses properly 
formulated across 2- 
3 domains. 

Problem and 

hypotheses 

properly 

formulated across 
2 domains. 

Problem and 

hypotheses 

properly 

formulated across 
1 domains. 

Problem and 
hypotheses 
improperly 
formulated. 

Significance/ 

contribution 

Outcome can 
actually be used in 
society directly. 

Outcome needs to 
be adapted slightly 
before it can be 
used in society. 

Outcome needs 
major revision but 
can be adapted 
before it can be 
used in society. 

Outcome is purely 
adoption of an 
existing solution. 

Novelty 

The outcome is 
original in relation 
to the existing work 
locally and 
internationally. 

The outcome is 
original in relation 
to the existing 
work locally. 

The outcome is 
original in relation 
to the existing 
work in the local 
community. 

The outcome is not 
original. 


The first three criteria are part of what we perceive as contributing to meaningful and sustainable user 
experience. The subsequent three criteria are research-based. Koster’s (2014) theory of fun, analogizes 
people as amazing pattern-matching machines. Once we recognize a pattern, we seek to see it recur. He 
further observes that “Fun is the process of discovering areas in a possibility space” and that “We talk so 
much about emergent gameplay, non-linear storytelling, or about player-entered content. They’re all ways of 
increasing the possibility space, making self-refreshing puzzles”. Moreover, for games to be art, they need to 
be puzzles with more than one right answer, open to the interpretation of the target audience. The challenge 
increases in multiplayer games where there are many instances with variations in player-entered content. 
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Elaboration on these metrics is explained in an extended version of this paper, which illustrates the 
evolution of our studies in the creative industries, contributions and implications, and the refined research 
model. Another paper compares these findings with the findings from the design of assessment for two 
Science-based courses. Software design and testing (Lee, Wong and Lau, 2015) and Robotics (unpublished 
technical report), its research model and the integrated Arts-Science model. 


2.3 Summary of Findings 


From prior studies, we propose an initial conceptual framework to develop generative designs. Our research 
model (Figure 2) considers the following: 

a) learning environments and systems need to focus on theorizing and high-fidelity prototyping; 

b) theorizing can be mediated by framing students’ role as active participants of and contributors to a 
larger intellectual and social community to create intrinsic motivation. This is because the situative 
perspective contextualizes the design of reflective scaffolds to build on prior knowledge triggered by the 
need to design with authentic future applications in mind; 

c) focusing on framing and reframing the context/problem in order to derive more alternatives and 
better solutions. This is based on the grounds that hypotheses are best formulated and tested within 
contexts, driven by the formulation and reformulation of goals/objectives; 

d) to sustain theorizing, learning needs to be fun and interactive; 

e) high-fidelity prototyping is essential during the theorizing, enactment phase; 

f) to be creative, technology needs to allow students to view a problem from different perspectives and 
abstractions to create more novel insights;. 

g) to develop multiple perspectives, we need to extend beyond functional designs to designing user 
experience; 

h) design thinking and metacognitive reflective scaffolds as embedded formative assessments need to 
be integral during lecture and tutorial; 

i) instructors must be brave to allow and look forward to emergent outcomes. 



Figure 2. Research model for this study 
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2.4 Conclusion 

According to Brown (2008), thinking like a designer can transform the way we develop products, services 
and even strategy. In our series of investigations, our objective is to develop a disposition for Social 
Innovations. 

To achieve this, we have aimed to: 

1 ) investigate whether our assessment rubric can be a means to motivate students/designers to learn, to 
model, to analogize, to theorize and to transfer learning through and to different contexts; 

2) develop an ontological affective-socio-cognitive model for context-aware interaction which cares - one 
which will intrinsically motivate and self-direct them to be more open, critical and creative. 

Subsequently, we hypothesize that if we utilize design thinking as our framing methodology, students are 
more likely to emphasize on user experience above functional requirements. However, our sample size is 
small. Hence, findings are not generalizable at that stage. As such, we do not make generalizable claims 
regarding the effectiveness of our scaffolds. At this moment, we can only claim that based on evidential 
findings, our scaffolds (design thinking, metacognitive reflection as embedded assessments and framing), 
help students to be more aware of the context and its relation with their prior knowledge, helps them to think 
broader, make more hypotheses and consider more alternative solutions. 

With this paper, we end our research into the creative industries. We hope to test our initial conceptual 
model in different domains, via different frameworks, interaction platforms and different devices, with 
diverse interdisciplinary collaborators, especially with regards to Science, Technology, Engineering and 
Mathematics and its variants. 

We are keen to collaborate with others within the gamification entrepreneurial framework, continuing 
with our investigations into Web modelling and engineering for the Sciences (Software design and testing) 
and Web modelling and craft for the creative industries while focusing on user experience. UPenn’s 
eCrafting.org and the Mixed Media Lab are referred to as examples. Possible areas of collaboration are: 

a) Enhancing youth's digital lifestyle through social-media-mediated co-creation [Media and 
engagement] 

b) Gamification to improve quality of life to the disabled (visual, learning, auditory). [Assistive tech] 

c) eCrafting for fun, convenience, lifestyle [DigiStyle tech] simulating UPenn’s ecrafting. 

d) Improving comprehension of programming language concepts through the integration of games. 
Lego and programming through visual functional gamified co-design [STEM] 

e) Technology-assisted affective-socio co-design as driver for e-commerce [Media and engagement] 

Ultimately, we hope that our findings in discovering an ontological Affective-Socio-Cognitive Co-Design 

Model and in developing a disposition for Social Innovations can be used to develop a Symbiotic 
Context- Aware Recommender suitable for IoT or Cloud Computing. 
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ABSTRACT 

A ‘bush kinder’ is the Australian equivalent of a European forest kindergarten. Although it is not usual for technology to 
be used in the type of program, the authors suggest that mobile technologies can be used creatively and sensitively to 
support learning in the bush kinder context. This paper describes an ethnographical case study where two early childhood 
researchers participated in a bush kinder program in “active participant -observer” roles and used a mini-iPad with cellular 
data access to extend investigations with the children. The advantages of this approach are that information about the 
natural environment can be accessed ‘just in time’ and experiences can be captured for future reflection and presentation 
to a wider audience. 


KEYWORDS 

Mobile technologies, kindergarten, nature education, bush kinder 


1. INTRODUCTION 

In his book, ‘Last Child in the Woods’, Richard Louv (2005) coined the term ‘Nature Deficit Disorder’ to 
describe how human beings, particularly children, are losing touch with nature because they spend 
progressively less time outdoors in natural spaces. The consequence of this disconnection is that children lose 
the capacity to restore their own health and wellbeing through relaxing in the natural world. Further, these 
children are likely to have a reduced capacity to develop an affinity with nature and therefore do not acquire 
responsible behaviours that are essential for ongoing environmental care and management. 

A common assumption of the ‘getting back to nature’ movement is that children should forgo their 
fascination with modern technologies, such as computer games and television, in order to interact with the 
environment in a pure state. In this paper, however, we question the premise that technology and nature are 
diametrically opposed. Rather, we see technology as a tool that, when used creatively, can actually help 
young children to make connections in the bush kinder nature setting and then be used as a medium for 
sharing their understanding and ideas about the bush when they return to their lives in a urban context. 


2. THEORETICAL FRAMEWORK 
2.1 The Nature Education Movement 

A renewed interest in connecting children to nature is being felt across the early childhood educational sector 
around the world. As far back as Froebel, the “Grandfather of Kindergarten”, in the early nineteen hundreds 
(Ebbeck and Waniganayake, 2010), preschool was seen as the child’s gateway to nature. This connection, 
however, faded during the corporatisation of early childhood environments in the nineteen eighties and 
nineties, when cement, artificial grass and synthetic ground cover became the dominate surfaces of play in 
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early childhood settings. As the first decade of the new century approached, it was noticed that children were 
losing important affiliations with the natural world around them. 

While children usually oscillate between indoor and outdoor play activity, it was noticed that children 
were spending progressively more time indoors and less time outside (Clements, 2004). This shift from 
active outdoor play towards a passive and inactive lifestyle generated concerns about the health of children. It 
was identified that childhood obesity was an increasing issue (Ellaway et al, 2007) and the physical fitness of 
children and the development of motor skills were in decline (Fjortoft, 2001). Another concern was that 
children were engaging less in creative, imaginative and make believe games (Clements, 2004; Fjprtoft, 
2001). It was also noted that when children did participate in outdoor activity, it was more likely to be in an 
organized sport that was heavily managed by adults rather than freeform outdoor play (Clements, 2004). This 
meant that rules and structure were imposed on children rather than being child-initiated and negotiated. 

The benefits of outdoor play for contemporary children have been well documented. Davis et al (2010) 
identify that health and wellbeing goals for young children can be clearly linked to “green” outdoor play. 
This play gives children the opportunity to develop physical skills, such as strength and flexibility, and 
physical self-awareness (Clements, 2004). Additionally, risk-taking in a natural space allows children to 
extend both physical and mental boundaries (Sandseter, 2009). Kellert (2005) also identified that play in a 
natural space supports the development of self-esteem and helps to build resilience against stress and 
adversity. This was reinforced by Mailer and Townsend (2006) who suggested that hands-on contact with 
nature could benefit the mental health and wellbeing of children. 

It has also been suggested that interactions with nature are beneficial for cognitive and social 
development (Strife & Downey, 2009). Sustained play in a natural setting is said help to improve 
concentration and facilitate the process of learning (Wilson. 2012). Further, unstructured play with 
improvised materials supports the development of imagination, creativity and cognitive development in 
general (Wilson. 2012). Playing with other children in an outdoor, natural setting offers further opportunities. 
Dowdell et al (2011) suggested that unstructured imaginative play in nature supports the development of 
positive relationships, provides a platform for social negotiation and helps children to learn skills and 
strategies for cooperation. 

A key premise for nature play is that a meaningful and connected understanding of the natural 
environment is an essential precursor for young children to develop a “stewardship of the planet” 
perspective. Davis et al (2010) identified that sustainability education is fundamental in order to engage 
future adults with the rising issues of climate change and environment degradation. This was reinforced by 
Ernst and Theimer (201 1 ), who made tangible links between a child’s early engagement with nature and later 
life environmentalism. Sustainability education can lead to short-term activism where children ensure that 
they don’t waste water, recycle rubbish etc. but it is also likely to have more far reaching outcomes (Lester 
and Maudsley, 2007). Logically, it seems to make sense that children who experience and develop an affinity 
with nature are more likely to develop long-term responsible environmental behavior that will protect the 
environment for future generations. 

2.2 Technology and Nature 

About the same time as outdoor and nature play went in to decline, the use of technology by children for 
entertainment rose to prominence. This phenomenon is seen as a direct connection in the literature. For 
example, McCurdy et al (2010) reported that the health and fitness of children declined as they spent more 
time indoors using electronic media and less time engaged in outdoor unstructured play. Larson et al (2011) 
identified that electronic media consumption, including internet use and texting, was the most common 
reason children gave for not spending time outside. This disposition is embodied in a well-known Richard 
Louv quote attributed to a 4 th Grader: “/ like to play indoors better ‘cause that’s where all the electrical 
outlets are ” (Louv, 2005). 

While it is likely that time spent inside using computer games and entertainment technology does 
encroach on time that could be used by children to engage in outdoor and nature play, it is important that the 
focus remains on the actual activity rather than the technology per se. Computer games and other electronic 
entertainment technologies such as Wii, Nintendo and iPad apps can be incredibly captivating and. while this 
activity arguably can have merit in regards to skill development, persistence and problem-solving, concerns 
relating to the time consuming and addictive nature of these pastimes are also valid (Plowman et al, 2010). It 
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does, however, needs to be recognised that research in the area of the use of technology by young children 
identifies that this sort of activity is only one in a spectrum of applications. 

Digital technology has revolutionized how children learn and interact with their world and there are 
plenty of instances where technology is used to extend learning opportunities and facilitate new ways to 
engage, create and communicate (Plowman et al, 2010). Di Bias et al (2013), for example, described a digital 
storytelling project in Italy where kindergarten children collectively author digital stories to communicate 
ideas about a shared experience. In a similar approach, Leinonen and Sintonen (2014) identified that media 
technology is a considered an everyday resource in the Finnish kindergarten where they supported children to 
use technologies such as digital cameras and audio recording devices for media production. 

Typically the ‘nature education revolution’ calls for a reversal of the technology/outdoor play nexus. That 
is, children need to put down the technological devices and go outside to interact with nature. To some extent 
this philosophy is reflected in the ‘no toys, no tools’ approach that is implemented in nature schools in order 
to connect children with the natural objects in the environment. Is this either/or approach the only option 
available to children? Technology, by definition, is the process of inventing tools and mechanisms that 
support humans to interact with the world around them. If we look at people who interact with nature in 
significant ways, either for leisure, such those who undertake wilderness trekking, or professionally, for 
example, scientists who participate in expeditions to the Antarctica, they don’t actually forgo technology to 
do this. What they do is select and use technology that is carefully designed and purpose built for the task. 

Is it possible that we can use technology in a natural outdoor environment in ways that demonstrates 
authentic application to young children? In a project with slightly older children, Williams et al (2005) used 
mobile technologies to create “soundscapes” in outdoor environments. In the project, the children used 
digital recording devices to create a sequence of sounds on a mobile device that would trigger from sensors 
as the device was moved around an outdoor space. Curnbo et al (2014) engaged children to design a digital 
resource based on an outdoor play space. In doing this they required the children to identify key features of 
the environment to reproduce in a fantasy themed digital game design. Cumbo et al identified that their 
motivation for this project was to “get the children ‘off the couch’ and outdoors into their local natural 
places” (np), therefore establishing a connection between the natural and online worlds. 

In the project reported in this paper, we investigate how educators and children can use mobile devices to 
access ‘just in time’ reference material during a bush kinder session. We propose this as an authentic use of 
technology that supports the core philosophy of nature education and therefore demonstrates that technology 
and nature education are not necessarily opposed. 


3. RESEARCH DESIGN 
3.1 Methodological Approach 

The discussion presented here is framed by a case study and uses an ethnographical approach, informed by 
participant observation in order to consider how the use of technology might support the goals of a 
nature -based kindergarten. Brewer (2000) identified that Ethnography is a qualitative style research that 
requires the researcher to establish a close association and familiarity with the context. While ethnography 
doesn’t necessarily require actual involvement in activity, it is usual for the researcher to be situated within 
the context in the role of ‘participant observation’. This methodological approach was a natural fit for this 
study because the two researchers, both qualified and experienced early childhood educators, had expertise in 
the context. One researcher specialised in nature play and sustainability education while the other had 
research interests in the applied and creative uses of digital technologies with young children. Both 
researchers had an active interest in the bush kinder program and had already established connections with 
the kindergarten, the teachers and the families of children in the program. 

For the research described in this paper, the two researchers visited the case study site on several 
occasions. A primary goal was to consider how the technology might be used to support the program but to 
also act in the role of a teacher/mentor to use technology devices with the children when relevant. This 
involvement equated to an “active participant-observer” role (Johnson et al, 2006) because of the authentic 
nature of the interaction. 
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3.2 The Setting 

In Australia the term ‘bush’ is commonly used to describe natural bushland or forest. Consequently the 
Australian equivalent of a nature kindergarten is a ‘bush kinder’. The bush kinder reported in this study is 
aligned with a conventional kindergarten program in the outer suburbs of a regional city in Victoria. The 
bush program commenced with a successful pilot program in 2013 and was then implemented fully in 
subsequent years. The bush kinder children attend the kindergarten at the site on one day for three hours of 
fifteen hours of kindergarten attendance per week. The setting is located on what was once a rural school site, 
owned by the Victorian Education Department. It is approximately eight kilometres from the kindergarten 
and is almost three hectares of natural bushland bordered by small farms. 

The sessions are held each week on location. Parents deliver their children directly to the site and then 
return to collect them at the end. Early childhood educators conduct the sessions, with an adult to children 
ratio of approximately 1:5 at each session. The old school building is still on the site in a clearing in the 
middle of the bushland. however, the kinder group does not access this building at all and the activity is 
situated entirely in the surrounding trees. At first glance the site seems to be undistinguishable from any 
other section of bush, however, the children are acutely aware of the physical boundaries of their 
‘classroom’, which are marked with fallen logs, key trees and a natural parapet. The ‘entrance’ to the site is 
guarded by a ‘portaloo’ (temporary toilet) perched on the escarpment and the children know that they must 
not pass through this gateway unless accompanied by the adult that collects them at the end of the session. 

The bush kinder program is managed by a sub-committee who work with the educators to determine the 
philosophy and curriculum. The children bring drinks and snacks but are not usually allowed to bring toys or 
other equipment in to the setting. The educators provide a large tarpaulin for the children to sit on but in 
keeping with their ‘no toys, no tools’ philosophy, they rarely have other teaching tools or materials. 
Additionally, they don’t allow the children to take any natural objects away from the setting. If materials are 
collected in the environment, they are left there for the next visit. Finally, the children need to be prepared 
for all weather conditions and bring warm clothes or wet weather gear if the weather is inclement. The only 
time bush kinder is cancelled is when it is very windy and there is a danger of branches falling. 

3.3 The Study 

The study described in this report was conducted over a period of 12 months, spanning two kindergarten 
cohorts as data was collected in the second half of one year and the first half of the next. In 2013 (the pilot 
implementation), the bush kinder cohort consisted of 13 children in their kindergarten year (aged 5). In 2014, 
the cohort was expanded to 26 children. In both years the children attended bush kinder on a Monday 
morning for three hours from 9.00am to 12.00pm. The minimum adults in attendance for the pilot cohort 
were three, two educators and one volunteer helper, while the larger group had three educators and two 
volunteers. On occasions more adults participated in a session, for example when the researchers visited, 
however, the teaching team were very careful not have too many adults interacting with the children. 

As described in 3.1, the two researchers visited the bush kinder sessions together or separately a number 
of times. On two of these occasions, a videographer also attended to film child activity and/or researcher and 
child interactions. For the other visits, the researchers took their own recording devices, a digital camera 
and/or a video camera to record activity when required. The intention of these visits was to collect data for 
several purposes, including immersing the researchers in the bush kinder culture, collecting footage for 
teacher education resources, observing the work of the educators in the program and, as presented in this 
paper, the application of mobile technologies in a nature education setting. 

The bush kinder educators hadn’t previously used technological devices in the setting, however, for the 
purpose of the study, the lead teacher asked permission from the bush kinder committee to allow the 
researchers to bypass the ‘no tools’ policy to use some technology in the setting. On two occasions the 
researchers took a mini-iPad with cellular data access to the site in addition to the digital camera and the 
video camera. The researchers also provided two magnifying glasses for children to examine specimens. The 
researchers were free to interact with the children as they saw fit and any learning moments using the 
technology were spontaneous rather than planned. Typically, a researcher would initiate an activity with a 
child or a group based on a topical interest or an observation and then children might join or leave the group 
as they wished. These learning events were recorded, either by the videographer or the second researcher. 
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4. FINDINGS 

Two vignettes of learning events where the mini-iPad was used to support authentic investigations of nature 
in the Bush Kinder setting (see Figure 1) are presented in this section. 



Figure 1. Using the iPad in the bush kinder setting 


4.1 Let’s Look It Up! 

In the first learning event, a group of three children had found a partial eggshell under a tree. The children 
were excited by their find and were imagining what animal might have hatched from the shell. One of the 
researchers scaffolded their discussion as they cycled through possibilities. The children quickly came to the 
conclusion that it would belong to a snake, because it was found on the ground. The researcher suggested to 
the children that they could use the iPad to look up pictures of snake eggs to confirm or refute their 
hypothesis. The second researcher joined the conversation, as they looked through the images. 

Child 1: This is a snake egg 

Child 2: 1 think the snake might be near! 

Researcher 2: What have you found? 

Researcher 1: We found this (shows the eggshell to Researcher 2), and everyone is being so gentle with it, 
and we were just trying to match it to some pictures. Do you think that is a fairly close match? 

Child 3: (holding a down feather that he found) Look at this! Do you know which bird this coined (sic) from? 
Researcher 2: Do you think it is from a bird or a reptile, a snake? 

Researcher 1: At this stage (Child 2) thinks it might be a snake, but we aren ’t sure, so let’s have a look at 
bird eggs. 

The group google ‘bird egg identification’ and find a page from the Museum of Victoria “Identifying 
birds eggs”. They recognise that the found eggshell is far similar to the bird eggshell pictures. 

Researcher 1: We have to wait a while for the Internet to load. Here we go... 

Child 1: Oh no. Too different. 

Researcher 1: Is our shell anything like that (Child 2)? Can you see any similarities? Have a look at the 
patterns... What about these ones... Are there any here that look similar? 

Child 1: No... Wait, wait! That one. 

The children and Researcher 1 looked carefully at the picture. She showed them how to use their thumb 
and finger to zoom in on the page. The children all had a turn at operating the zoom in mechanism. The 
children decided, however, that the picture didn’t match their egg. They continued to look until they find a 
picture of a Tawny Owl egg that was almost identical. They read the information on the Tawny Owl and 
found out that it is a nocturnal bird and lives in the local area. The children then decided that the eggshell 
should be placed in the ‘Bush Museum’, an arrangement of found natural objects displayed on a tree stump. 
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4.2 Identifying Plants 

In the second learning event. Researcher 1 worked with two boys in the group and they were examining 
flowers under the magnifying glass. She again used the mini iPad with them to identify the specimen. 
Researcher: We want to find out what this flower is called. (Child 2) will you come over and help me please? 
OK, guys, do we have the specimen handy? ( Child 1 ), can you put it under your magnifying glass? (Child 2), 
are you having a good look? Now, I’m going to show you some pictures... 

Child 1: Then we can see what we can call it... 

Researcher: Yeah, we want to find its real name. 

Child 1: So we can call it something. 

The two boys look carefully through the magnifying glass and the researcher points out the various parts 
of the flower. They look at the images on the ipad again. 

Child 2: 1 think it looks like that. 

Researcher: This one? What do you think (Child 1 )? 

Child 1: 1 think it is that one (points to another flower). 

Researcher: Do you ? OK. We need to investigate. 

Researcher: OK, let me tell you a bit about this one. It is called the Prickly Wattle and it blossoms in the 
spring, which is now. 

The group looked through the information displayed about the Prickly Wattle and decide that their 
specimen matches the photograph and description. 

Researcher: OK, are we happy with this name ? 

Child 2: Yes. 

Researcher: Alrighty. I want to have a look at the other one you spotted. Where was that? 

Child 2: Over there. 

Researcher: OK, lets go. (The group walks through the bush) 

Researcher: Where is it? (Child 2 shows the researcher) 

Researcher: Wow it is too. And that is different! What is different? 

Child 1: It is different because it has big leaves! 

Researcher: Did you notice that too (Child 2)? The leaves are quite different. The leaves on our other Wattle 
were quite small and these leaves are large. And it is certainly higher, it is taller! 

Researcher: So lets see what we can find. ..the leaf is quite different on this one isn ’t it? 

Child 1: But the same model! 

Researcher: Yeah. Similar flower, different leaves. 


5. DISCUSSION 

The two vignettes presented here show that it is viable to use connected mobile technology as an 
investigative tool to solve authentic problems in a natural setting. In these scenarios, the ipad were used by 
early childhood educators (the researchers) to engage young children in scientific investigation processes. In 
these learning events, the children were asked to wonder about objects they found and hypothesise about 
where they might belong in the natural world. They were also required to examine the characteristics of the 
objects and compare and contrast to online photographs for classification purposes. In these conversations 
they also learnt about natural processes such as life cycles and seasons and they used nomenclature and 
scientific terminology such as ‘specimen’ and ‘similarities’. 

The first learning event was instigated by a discovery, the partial eggshell. The children assumed that the 
egg belonged to a land animal (a snake) because the eggshell was found in leaf litter and so the ‘just in time’ 
information provided on the iPad helped the educators to address this misconception and steer them in a 
more appropriate direction of inquiry. This activity also gave the educator an opportunistic moment to teach 
the children more about using the iPad and supported the children to practice physical skills required to zoom 
in and position pictures on the screen. Additionally, the event helped to highlight authentic connections 
between the Museum of Victoria, the source of the information, and their own Bush Kinder Museum. 
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The second learning event was more devised by the educator (Researcher 1). In this activity, she 
addressed the scientific practice of naming species with the children. Rather than simply identifying all 
flowering plants, as ‘flowers’, or by colour, ‘yellow flowers’, she used the iPad connectivity to introduce 
early concepts of biological classification and to help the children to differentiate between similar species. It 
was interesting to see that in this vignette the children didn’t pick or even touch the flower they were 
examining. They examined the specimens in situ and then used the iPad to photograph the plants for future 
reference. 

As described previously, these two events were spontaneous interactions between the children and the 
researchers rather than planned curriculum activity by the bush kinder educators. They do, however, provide 
a useful insight in how mobile technologies may be used to support the early childhood curriculum in a 
nature education setting. Not only can this type of technology use support just in time investigations by 
giving children and educators access to online information, it is possible to use the technology to capture the 
moment for extended reflection. The educators could bookmark the relevant websites at the time in order to 
revisit with children later in the week at the conventional kindergarten. They could also photograph the 
objects, video the children’s interactions or record on the spot reflective observations for later reference. The 
Bush Kinder program has a policy where objects are not removed from the location and so digital collection 
is a way to build connections and extend learning back in to the kindergarten sessions. Further, the children 
from the bush kinder could use digital media to represent their own stories about bush kinder. This will also 
enable them to reflect on their experiences in the setting and share them with family and friends. 

An added advantage of using technology in the bush kinder setting is that educators can also use it to 
reflect on their own teaching practices. During the ‘Let’s look it up’ learning event while the group were 
debating whether the egg was from a bird or snake, one of the children (Child 3) was holding a down feather 
that he found in the same location as the eggshell and asked ‘do you know what bird this corned (sic) from?’ 
Neither of the researchers noticed him at the time, but the feather was distinctly from an owl and if his 
contribution had been included in the conversation, it would have been a valuable clue to the identity of the 
eggshell ! It is important that early childhood educators are informed about the natural environment they are 
working in order to support the children they are working with (Grogan, 2014). The use of technology to 
record interactions in the bush is an effective way to raise awareness for educator reflection on teaching 
moments and curriculum planning. 


6. CONCLUSION 

The ‘no technology’ position taken by nature education advocates stems from some fundamental concerns 
about sanctity of childhood and the diminishing connection that children have with nature. Technology is 
seen as an unwelcome disUaction for children as they are sucked in by pervasive entertainment technologies 
that are mind numbing and addictive. We respect this concern and also lobby for the ‘back to naUire’ 
movement where children can explore and connect with the natural environment. We, however, believe that 
games and elecUonic entertainment offer a very narrow perspective of technology and it is important that 
educators recognise the powerful ways in which the broader spectrum of technology can be used to support 
learning in any context. We consider that technology, and particularly mobile technology devices such as 
iPads can provide exciting possibilities in a nature education context and we are truly grateful for the 
opportunity to explore this avenue in our bush kinder. 
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ABSTRACT 

The higher educational landscape nationally and internationally has changed over the last decades, moving from an elite 
to a mass higher educational system (Trow, 1984). Currently this system and the organisations within it are contending 
with new external influences on their future structure — the arrival of the technological revolution and digitalisation of 
learning. Technology has taken global access to our living rooms. Many embrace digitalisation of learning, but others are 
not so positive. Using identity theory, we examine the potential impact of this shift within the Institute of Technology 
sector in the Republic of Ireland with a particular emphasis on exploring the impact of learning technology on 
organisational and academic identity, concluding that exploring this impact is the first step on the journey of digitalisation 
of learning. 
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1. INTRODUCTION 

The purpose of higher education is the belief in individual and societal value, where theorists cite benefits 
including economic returns, improved life outcomes, and opportunities for individuals (United Nations 
Educational, Scientific and Cultural Organization (UNESCO), 1991). This is supported within the Republic 
of Ireland as is evidenced in the most recent higher educational landscape strategy, where it concludes: 

“Higher education is central to the economic renewal we need to support individual well-being and 
social development. . . The quality of their learning experiences and the environment in which students 
learn will shape the future development of our society. They are also citizens who will add to the richness 
of society - as parents, community leaders and teachers- and in their chosen area of work they will be the 
productive engine of and prosperous economy” (Higher Education Authority (HEA, 201 1; p. 9). 

The impact of this agenda on individual higher educational institutions is the requirement to provide 
programmes of learning that meet the needs of society and the learner. This has led institutions to reflect on 
the business they are in, how they are doing, what they are doing, and should they be doing something else to 
achieve the above. Such reflections have led to the realisation that the space higher education exists in has 
shifted and will continue to shift in the future, where Ireland’s higher educational system must become more 
flexible in provisions of both time and place and one that facilitates transfer and progression through all 
levels of the system (Higher Education Authority (HEA), 201 1). 

With this comes the understanding that flexibility and accessibility of provisions are limited within this 
space, where programmes of learning are primarily delivered face to face within specific institutions (HEA, 
2011; p. 38). Recently, however, there are isolated examples of programmes that are available on a flexible 
online basis. This use of digitalisation to make learning more accessible is advocated at government level, 
where the sector undergoing transformation concludes that in “the years ahead, students will choose to learn 
in a variety of ways- full-time or part-time; on campus or off campus; class room based, blended, online or 
accelerated learning” (HEA, 2012; p. 54). 
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Therefore, the question has emerged as to whether institutions are ready for this shift into flexible and 
more accessible learning, which represents a significant cultural and operational challenge. Are they able to 
reshape themselves to embrace rather than rebuff this agenda? (McCarthy and Samors 2009). This is tandem 
with our children who are growing up as digital natives (Presnky, 2001; National Association for the 
Education of Young Children (NAEYC), 2012) and as such will support the notion of a digitally based 
pedagogy even from a young age (McTaggart, Cavaliero, and Pender, 2014). With some theorists concluding 
that not only will it improve the flexibility and accessibility of higher education, but it will also ensure that 
today’s teaching and learning is delivered in a way that is relevant to today’s population by offering the 
potential to reach out and engage ‘the other half of the young population as well as to reach out to adults 
who have not previously benefitted from higher education. Therefore, the belief is that this will provide a 
fairer, faster higher educational system that is more in tune with contemporary society (O’Connor, 2007; 
Selwyn, 2014). 

Evidence would indicate that, even within this acknowledged potential, concerns exist regarding the 
impact of digitalisation on the notion and legitimacy of higher education institutions. This paper examines 
these concerns by exploring the impact of digitalisation on the identity of both the organisation and the 
academic, with the goal of providing a knowledge base that will help to successfully manage this shifting 
space within the Institute of Technology (IOT) sector in the Republic of Ireland. As the stronger an 
organisation’s identity, the better their knowledge, attitude, and behaviour regarding the achievement of 
organisational objectives. 


2. LOCAL CONTEXT 

The Republic of Ireland’s higher educational system has been loosely considered a binary system (two higher 
education institution types) in that the majority of provisions exist in the traditional unified university sector 
of which there are seven or more applied and career-focused IOT’s (14) (Mulcahy, 1981; Lillis and Morgan, 

2012) . While they still have an underlying core ethos of vocational and technical programmes from their 
originations as regional technical colleges, the IOT sector is now similar to their university colleagues, as 
they too have the authority to deliver degrees at undergraduate, masters and doctorate level. Some can do this 
independently; others require validation by the Higher Education and Training Council (Taylor, Brites 
Ferreira, de Lourdes Machado, Santiago). In addition to the main higher education providers above, a number 
of other third-level institutions deliver specialist education in fields such as art and design, medicine, 
business studies, rural development, theology, music, and law. 

From a geographical perspective, three of the universities are located in the capital city, Dublin, an 
additional university is in Co. Dublin, and the remaining three are located in the largest cities in the country: 
Cork, Limerick, and Galway. Comparatively, IOT’s (similar to historical poly techs in the United Kingdom) 
are more regional in focus, located often in areas of higher unemployment and social disadvantage (Burns, 

2013) . 

Participation in higher educational programmes of learning has risen steadily from a base of full-time 
enrolment in 1967 of 21,000 to its most recent figures of over 160,000 full-time enrolments (Orichesta 
Research and Library Service, 2014). This indicates that 50-55% of 17 to 18 year olds enter higher education 
(O’Connor, 2007). In a breakdown, the IOT studentship makes up almost half of the higher educational 
provision with University studentship numbers at just under 77,000 and IOT’s at just over 73,000 (Orichesta 
Research and Library Service, 2014). 

While these figures are positive, the profiles of the learner groups who attend particular higher education 
institutions are different. The university sector has traditionally espoused the values of elite higher education, 
while the IOT’s are pluralist and as a such are more reflective of a mass higher educational system, where a 
significant proportion of their student cohorts are non-traditional and/or are the first generation to higher 
education (Mooney Patterson, O’Connor, and Chantler, 2010). Within this frame changes are underway in 
the IOT sector, where if these institutions meet a number of key requirements, including capacity (critical 
mass student numbers and future protections), capability (staff qualifications and research), and expertise, the 
HEA has committed to allowing them to apply to become a technological university. This technological 
university is intended to be: 
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“(Internationally, a technological university is) a higher education institution that operates at the highest 
academic level in an environment that is specifically focused on technology and its application” (HEA, 2011; 
p. 103). 

Meeting the criteria of capacity and critical mass will require institutes of technology to merge often with 
significant geographical distances between. Consequently, the role of digitalisation not only supports the 
needs of modern day learning and non -traditional study arrangements (part-time, e-learning, etc.); but also 
supports a restructure of the sector, which is now at the forefront of institutional agenda. This is not 
necessarily to allow for programme rationalisation, but to allow students accessibility to programmes from a 
wider campus than may be possible in their own regionally based campus, thus meeting the capacity 
requirement (HEA, 2011; Marginson Report 2011; p. 28) and ensuring fit for purpose in a rapidly changing, 
globalised world. 

A small number of institutions have embarked on this journey and have submitted their expression of 
interest. Some have progressed to Stage 3 i.e. Evaluation of Plan stage of this process; however, concerns 
still exist within the entire IOT sector regarding what this restructure and these requirements will mean to all 
key stakeholders inclusive of the organisation itself and its academics. Anecdotally, the part that appears to 
evoke significant fear is the role and the method that digitalisation of learning may play in restructuring of 
the sector. 


3. IDENTIFYING AND IDENTITY 

While research clearly indicates that digitalisation will not cause the disappearance of campus-based teaching 
and learning, it will transform the way education is delivered and supported (Economist Intelligence Unit, 
2008). From an organisational perspective, the principal direct impact of digitalisation and learning 
technology is the scope to significantly change how higher education delivers in terms of volume and 
distance. Specifically, the ability to reach a greater number of students with fewer resources and to reach 
students over much longer distances while maintaining direct interaction in real time through shared online 
spaces is possible (HEA, 2011; University UK, 2014). While this has its advantages in terms of critical mass 
and economic opportunities, the impact on how such a shift can cause confusion to existing organisational 
identity should not be ignored. Organisational identity is understood as shared beliefs about the 
organisation’s central enduring and distinctive characteristics (Bauman, 1996; Wiesenfeld, Raghuram, and 
Garud. 1999), which can enhance organisational effectiveness and performance and serve as a source of 
sustained competitive advantage (Barney, Bunderson, Foreman, Gustafson, Huff, Martins, Reger, Saranson, 
and Stimpert., 1998). This identity is shared by members and is created and developed through a continuous 
process of claims and counterclaims regarding the concept of what the organisation is ( Whetten and Godfrey, 
1998). This leads to the question, what it a university, and what is an IOT? 

By allowing and facilitating students to not attend a classroom and as such move to something that we 
cannot identify with in terms of the normative student experience where students are educated not within a 
physical structure, this challenges the traditional image of higher education, its distinctive characteristics, and 
its abstract and concrete identity (Reger, 1998). Abstract identity is understood as components that help to 
establish an organisation’s context and reflect their culture and values that transcend over time (i.e., why we 
do what we do in the way we do) as well as the legacy of it. Concrete identity elements are tied to a particular 
time and set of environmental conditions, product strategies, and environment scope (i.e., tied to the 
historical perception of higher education and elitism pluralism) (Stimpert, Gustafson, and Sarason, 1998; p. 
91). 

Additionally, within the IOT sector, abstract and concrete identities (Stimpert et al., 1998) are based on 
foundations and legacies from regional technical colleges, where it has held and continues to hold, the value 
and belief that they are local institutions who serve the needs of local populations. Consequently, this shift to 
technological university status requires a critical mass and an improved use of technology to support learning 
opportunities, which may challenge the notion of what is local, may be inconsistent with those expectations, 
and has the potential to create tension within the organisation and within the sector (Golden-Biddle et al., 
1997). Selwyn sums this up by concluding that digital technologies challenge and question the very essence 
of what is a university (Selywn, 2014). On this occasion, the introduction and use of digitalisation that 
deviates from the expectations associated with this organisation’s identity are challenging, as identity is 
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intertwined in the routines, procedures, and beliefs of both organisational and external bodies, and efforts to 
shift identity in order to accommodate identity-challenging technology are difficult to achieve (Tripsas, 
2009). 

Equally, organisations gain legitimacy with internal and external stakeholders when they demonstrate the 
normative identity of (in this case) higher education, but this can cause unease if it varies too much from 
what is expected from the organisation (Navis and Glynn, 201 1 ). 


4. WHAT DOES THIS MEAN FOR THE EDUCATOR? 

A key part of organisational identity is that of those who teach in that space and their academic identities, 
which (at the core) relate to teaching and research activities that they deliver (Deem 2006, p. 204). While 
much of this identity is informed by disciplinary specific knowledge commonalties in values and beliefs are 
believed to exist, these include but are not limited to academic freedom, the community of scholars, 
individual autonomy, and service to society through the production of knowledge, transmission of culture, 
and education of the young (Kuh and Whitt, 1986, p. 76; Whitechurch and Gordon, 2010). The introduction 
of any new change challenges and redefines some or all of these values: freedom, autonomy, and 
transmission of culture. None could be seen as more challenging than the introduction of technology to 
support digitilisation of learning (Adams, 2012), which ultimately impacts on their symbolic order, 
understood as “the form that the various species of capital assume when they are perceived and recognized as 
legitimate” (Bourdieu, 1989, p. 17). This order allows academics to recognise themselves in, or identify with, 
the image presented in the ideology and behave accordingly to who they are within a given context (Zizek, 
1989 pp. 43-44; Runions, 2000; p. 197). 

This challenge to identity is exacerbated by the generation gap of the technological revolution. While 
most who teach are of the generation of the digital immigrant, this is at sea to our digitally native students 
who currently use technology based devices earlier than they hold a pencil and often with more ease and 
proficiency (Lauricella, 201 1 ). Added to this is the reality that “academics form an unusually independent 
profession, one that draws heavily on traditional ideals about its proper role even as its present-day operating 
realities in many countries have become increasingly diverse” (El-Khawas, 2002). A tradition of 
traditionalism is asked to embrace something that is inherently different, which many feel impacts on their 
legitimacy; it is easy to see why anxiety can and does occur (Levy, 2003). This ultimately compels those 
delivering programmes to confront existing assumptions of teaching and learning in higher education 
(Garrison and Kanuka, 2004). 


5. IDENTITY REFORMATION 

While firms and (in this case) higher education institutes have a current understanding of organisational 
identity, they also have a desired or ideal concept of identity (Stimpert, Gustasfon and Sarson, 1998). This 
existence of an identity gap can be useful in a transformation process. On this occasion, members of the 
organisation and stakeholder groups can be supported to develop a conceptualised identity that embraces 
digitalisation in a newly configured technological university. Successful strategy renewals can allow an 
organisation to discard some concrete identities that are no longer of value, while simultaneously adding new 
concrete attributes that build on the abstract identity attributes and offer the potential of new advantages. 
Within this IOT transformation process this could include building on the abstract identity that accords the 
value of education and is entwined with the organisational belief system of making higher education more 
accessible to non-traditional groups. This philosophy can be supported through digitalisation’s ability to 
increase student capacity in a medium that is relevant for today’s population. This change will add a 
competitive advantage to the concrete identity, while preserving a number of unique characteristics (Gioia, 
1998; p. 22). 
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The concept of organisational identity, therefore, motivates actions as well as strategic choices (Ashforth 
and Mael, 1996) and is something that should be considered. Identity is who we are and the current and 
future strategy details with the actions to take us to who we are to become, “a generator for strategies, a 
screen, a constraint, a filter, an enabler and an influence on strategy” (Barney et al., 1998; p. 166). This is a 
lesson that can be learned in the current shift within Ireland. The key is to ensure communication and training 
to support the shift, while maintaining the symbolic capital that is imperative to support and protect the 
identity of the organisation and its members (Henkel, 2005). 

For those who have taken this leap, the new place has proven not that scary, recognising that this change 
is occurring and the reality is that if we do not embrace it, we run the risk of being left behind. 


6. CONCLUSION 

While the future higher educational landscape remains unclear, early research indicates that we will remain 
the same, but different than what we were. This will be the greatest challenge to accept — challenging the 
notion of our identity. This will provide institutions with the opportunity to contribute to a new identity 
formation, taking into consideration shifts that are inevitable due to the arrival of the digital age, to be viewed 
as an opportunity rather than a threat. We must imagine our future within this new future of the technological 
university where technology will be the key to its success, not demise. The potential of digitilisation to 
facilitate the accessibility of education to those who may not get this opportunity is one advantage, but 
another advantage is the potential of digitisation to change the way we do what we do. If considered a tool to 
support effective pedagogy, it can become less threatening and may have little impact on the identity of the 
lecturer. The future of who and what we can be should be the early focus of any restructure with the vision 
not constrained by what we can do, but inspired by possibilities and opportunities. 


7. RECOMMENDATIONS 

A next phase in exploring the impact of digitalisation of learning on academic identity within the IOT sector 
requires the collection of primary data in this field. This has commenced, where the author is currently 
undertaking an interpretive study with a number of participants who recently began their journey to digitalise 
their pedagogy and programmes within a case study institute. The study, through participants own narrative, 
examines their pre-knowledge and expertise in this area, their experiences before and during this process, any 
barriers or supports to this change in their pedagogy, its impact on their academic identity and any 
recommendations they may have to make this transition a success into the future. 

Early results indicate that the key to the success in this transition process is training, mentoring and 
support before such a change occurs, where participants concluded that such training reduces the anxiety and 
threat that this transition can bring to them and their professional identity. 
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ABSTRACT 

The purpose of this work-in-progress study is to examine science identity of elementary school students in relation to 
participation in science. The questions asked in this initial analysis were: How will a field trip to a science research and 
learning center affect students’ desires to learn and participate in science and does interviewing scientists alter this effect? 
A total of 151 elementary school (Grade 5) students (76 female and 75 male) participated in an experimental study. 
Participants were randomly assigned to two different experimental groups: 1) The experiment group attended the field 
trip and had the opportunity to meet and interview scientists. 2) The control group attended the field trip and did not meet 
and interview scientists. Results showed that all field-trip participants reported greater interest in their desire to become a 
scientist and participate in science after the field trip. Additionally, after the field trip, students who interviewed scientists 
were significantly higher in their desire to become a scientist than students who did not. 


KEYWORDS 

Role model, science identity, science participation 


1. INTRODUCTION 

How can we increase participation in science? A review of the literature provides an extensive list of factors 
thought to influence attitudes toward science. Determinants of who would like to learn science from a young 
age include gender, family, ability, youth culture, popular culture, education, support systems, personal 
interests, experiences, and time-use. 

Interest in science has been related to how students feel about what they are studying, the learning 
environment, and concept of self (Bloom, 1976; Hurd. 1978). Opportunities to develop a sense of being 
someone who does science (Aschbacher et al., 2010) are considered important to science identity. Research 
on dimensions of science-related interest indicates that students’ access, activity, achievement, and attitudes 
are key to interest in studying science (Hanson, 1996). 

Additionally, participation and a place of learning are considered to be of interest to students’ academic 
achievement and participation. Non-classroom activities are thought to enhance academic achievement 
(Campbell & Wolbrecht, 2006). Participation in science is generally related to a complex set of structural 
factors and contextual characteristics (McNeal, 1999). More specifically, participation in school 
extracurricular activities is thought to strengthen student ties to academic achievement, and opportunities for 
participation in science learning outside the school will provide additional opportunities for development of 
academic interests. Parents, mentors (formal and informal), heroes, and adults of interest have all been 
recognized for their influence on students’ sense of self and academic achievement (Pleiss & Feldhusen, 
1995). Finally, studies indicate that students express positive attitudes toward science when they experience 
success and receive support from others whom they consider to be important (Aschbacher et al., 2010). The 
current research presents an experimental study that was designed to compare the change in a student’s desire 
to become a scientist between students who attend an out-of school science-related field trip with those who 
are randomly selected from the group to also meet and interview working scientists during the field trip. 
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2. CONCEPTUAL RATIONALE 

2.1 Science Identity 

Identity is comprised of components of adolescent development that serve to form identity for occupation 
and support the solidification of person in Super’s (1980) life-span view of an articulated framework for 
vocational development. Merolla and Serpe’s (2013) study of science-related enrichment programs and 
science identity among 654 graduate students reported that while gender, race, socioeconomic status, and 
psychological factors affect probabilities of girls entering a STEM profession. Science identity (or self 
concept) magnifies the influence of academic achievement on enrollment in graduate university programs. 
The success of STEM enrichment programs are thought to be intertwined with the social psychological 
process by which students may come to self-identify as scientists (Carlone & Johnson, 2007; Egan et al., 
2012). Carlone and Johnson (2007) conducted an ethnographic study of science identity among successful 
women of color and reported that recognition of self as a scientist is an important component of success. 
Egan et al. (2012) drew from a study of 1,133 aspiring STEM college freshmen that emphasized the 
importance of the establishment of science identities at the earliest time. They reported that science identity 
promoted academic success by increasing the likelihood of participation in valuable activities and 
experiences, such as early research and time with professors, which would in turn nurture their STEM 
identities. 

Aschbacher et al. (2010) suggest that it is important to provide students with opportunities to develop a 
sense of being someone who does science. Possible selves have been linked to academic achievement 
(Oyserman et al., 2006). Evidence suggests that the number of academic possible selves declines across the 
school years and that “Low income, rural and Hispanic youth are at risk of having few academic or 
occupational possible selves, or having such general possible selves in these domains that they are unlikely to 
promote self-regulation.” (Oyserman & Fryberg. 2006, p. 1). It is therefore important to provide support for 
science interest to students in such at-risk populations. 

Possible science selves are metaphors of self for being a person of science. Future possible selves are of 
particular interest as socially unverified aspects of being that are relatively impressionable components of 
self-regulation, motivation, and behavior (Markus & Nurius, 1986). Science selves are of interest for gauging 
student attitudes toward science liking, science learning, and participation. Smith (2006) examined possible 
selves in relation to expectancy theory and posited that possible selves are the driving force within 
self-concept that “should play a central, systemic role in the operation (self-regulation) of self-concept on a 
day-to-day basis” (p. 60) and can therefore play a key role in a student’s participation in and learning of 
science. 

2.2 Role Models 

Sjpberg (2002) identified factors contributing to what he describes as disenchantment with science and 
technology (S&T) among young people today. He attributed the trend toward insufficient recruitment in S&T 
to contemporary youth culture within social and political phenomena that varies and is best described 
individually by country. Sjpberg offered a tentative list of rationale that relates to the larger social trends in 
participation in S&T. Three of the 13 items on his list relate to science stereotypes and role models: 1) The 
role models in many cultures are not employed in S&T. 2) Scientists and engineers are no longer heroes and 
are sometimes blamed for “evils of environmental degradation, pollution, global warming, etc.” 3) 
Stereotypical images of scientists and are as follows, “Scientists (especially in the hard, physical sciences) 
are by pupils often perceived to be authoritarian, closed, bored - and somewhat crazy” (Sjpberg, 2002, p. 4). 

A study that focused on providing middle school students with exposure to women in science found that 
students became more positive in their attitude towards scientists and the role of women in science after the 
exposure to women in science (Smith & Erb, 1986). It is suggested that teachers of science in the middle 
school/junior high grades should periodically bring community resource people who use science in their 
careers to the classroom to act as role models. Additionally, they recommended that all students should be 
included among the groups of professionals who visit students so that the attitudes of both male and female 
students toward scientists and women in science might be improved. 
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A review of the literature examining role models among gifted children reported gifted students benefited 
from relationships with successful adult role models and heroes within their areas of interest. They report that 
there are educational benefits for gifted children who have the opportunity to interact with and imitate 
behaviors of scientists (Pleiss & Feldhusen, 1995). 

2.3 Participation in Science 

A number of studies have focused on factors that create interest, encourage participation, and foster a sense 
of being someone who does science among young people. Research indicates that participation in science 
activities, such as library/museum visits, have a significant direct effect on science attitudes (George & 
Kaplan, 1998). 

Persistence in continued participation on the science-related track is also an area of interest because many 
students who are initially interested in science eventually decide not to stay on the science academic path. 
Aschbacher et al., (2010) examined students’ persistence in the science-oriented academic pipeline. They 
identified high, low, and lost groups by persistence, which were found to be different in terms of interaction 
and experience within three science communities: in school, outside of school, and within the extended 
family. Students who participated and found support for science in multiple communities were more likely to 
have science identities and to persist in science learning (Aschbacher et al., 2010). Persistent science interest 
and achievement has also been associated with having solid support for science interest at home, at school, 
and in extracurricular activities (Schneider & Stevenson, 1999). 

2.4 Research Questions and Hypotheses 

As previously mentioned, research indicates that participation in science learning activities outside of school 
can increase interest in learning science. Exposure to people of science (Smith & Erb, 1986) and receiving 
support for science interests, by people of interest, are both associated with the strengthening of 
science-related attitudes among young people (Aschbacher et al., 2010). Additionally, role models and adults 
worthy of imitating can influence students’ sense of self and academic achievement (Pleiss & Feldhusen, 
1995). Therefore, it is expected that students who attend the science field trip will become more positive in 
their self-concept for being a person of science. In addition, it is expected that students who also have the 
opportunity to participate in a small group discussion with and to interview scientists will have the most 
positive change in science attitudes as measured post experiment. In particular, the following hypotheses are 
proposed: 

Hypothesis 1: Students who attend the field trip will be more positive in their perceptions of possible 
science self for desire to become a scientist pre-post field trip. 

Hypothesis 2: Students who attend the field trip will be more positive in their perceptions of possible 
science self for participation in science-related activities pre-post field trip. 

Hypothesis 3: Students who attend the field trip and meet/interview scientists will be significantly more 
positive in their perceptions of possible science self for desire to become a scientist after the field trip than 
students who only attend the field trip. 

Hypothesis 4: Students who attend the field trip and meet/interview scientists will be significantly more 
positive in their desire to participate in science activities after the field trip compared to students who only 
attend the field trip. 


3. METHODS 
3.1 Participants 

A total of N= 1 5 1 primary school students took part in the study. The students who participated were enrolled 
in Grade 5 at a public school in the United States. Participants were almost evenly split by gender (76 girls 
and 75 boys). The participants in this study were part of a larger group of 324 students attending the field 
trip. Study participants were students for whom the following could be matched: 1 ) study permission slips, 2) 
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pre-surveys, and 3) post-surveys. These participants were all from the larger field trip group, which 
comprised the entire 5 th grade of one school district, excluding a small number of students staying back at 
their campus due to In School Suspension or because they failed to provide a field trip permission slip. 

3.2 Measurement and Instruments 

This study employed multiple methods to gauge students’ science identity for being a person of science. The 
following instruments were administered to test participants’ predispositions and changes of dispositions over 
time. These instruments were measured on a 5 -point Likert-Yike scale with a response scale ranging from 
strongly disagree to strongly agree : 1) Possible Science Selves (PSS; 20 items; desire factor Cronbach’s 
alpha=0.86; participation factor Cronbach’s Alpha=0.80; Beier et al., 2012). 2) Creative Tendencies (13 
items; Cronbach’s Alpha=0.84; Mills et al., 2011). 3) Career Interest Questionnaire (12 items; Cronbach’s 
Alpha=0.84; Bowdich, 2009). 

The question items for the PSS factor for desire (hope) possible self are as follows: 

• I have always hoped to have a job in science one day. 

• Having a job in science one day is very important to me. 

• I expect to go to college and get a degree needed for a job in science. 

• It is very likely that I will get a job in science in the future. 

• I expect to have a strong professional career in science in the future. 

The question items for the PSS factor for participation (strategic) possible self are as follows: 

• I want to take as many science classes as possible next year. 

• If possible, I would get involved with a science club. 

• If possible, I would want to attend a science camp. 

• I would be interested in participating in a science fair. 

In addition to participating in the experiment and completing surveys, students also drew a scientist and 
responded to open-ended response items to describe both a scientist and an engineer prior to attending the 
field trip. Preliminary findings from the field trip experiment are presented here. 

3.3 Design 

This is an early report from work-in-progress research on Who wants to be a scientist? conducted among 
students in primary school Grade 5. The relationship between an out-of-school science experience and 
science self-concept is explored. For the first phase of this study, students drew a scientist, listed 
characteristics of scientists and engineers, and completed pen and pencil survey instruments to provide 
responses to items of the PSS, Creative Tendencies, and CIQ instruments. These activities were completed at 
the students’ home school. The experiment to test the effects of meeting and interviewing scientists was 
implemented during the field trip. Students participated in the various field trip activities in two groups: 
control and treatment. Approximately one-half of the students were selected at random to receive the 
treatment, which was meeting and interviewing scientists. The treatment group viewed an introductory 
movie, met/interviewed scientists, toured the working control room of the science facility (with scientists 
present), participated in hands-on science activities, and explored the interactive exhibit hall in the facility’s 
science education center. The control group participated in the same activities; however, they did not 
meet/interview scientists. Instead, they met with a science educator who presented information on the 
research activities of the science facility and demonstrated aspects of physical science applied for the 
research. Students in the two groups were not aware of the difference in field trip activities. 

The treatment group of students participated in 30-40 minute sessions, during which small groups of 
students ( 10-20 students per group) met and successively interviewed three science-related professionals who 
are employed in the research work of the field trip site. The science-related professionals who met with the 
students were both women and men of varying ages from a wide range of geographic locations. The topic of 
the interview session between the students and the science-related professionals was becoming a person of 
science. Students were provided with a script of suggested interview questions. Discussion was not limited to 
the scripted questions. 
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3.4 Data Analysis 

For each participant, initial (pre-field trip) Creative Tendencies and Career Interest Questionnaire scale 
scores were determined along with matched, paired pre-post PSS desire and participation scale scores. Data 
were examined by analysis of means procedures pre- to post-field trip and between control and treatment 
groups. 


4. RESULTS 

Paired samples (-test analysis was conducted to compare students’ scientific self-perceptions for PSS desire 
and participation pre- to post-science field trip by control and treatment groups. Additionally, analysis of 
variance was conducted to compare differences between the experiment and control groups in PSS desire and 
participation factors post-field trip. 

4.1 Pre-post Analysis 

Control group analysis of (-test pre-post data (for students who attended the field trip only) revealed a 
significant difference (increase) on the academic desire PSS factor pre (M= 2.53, 573=. 939) and post (M= 2.82, 
573=1.04), ((75 )=-3.40, /?=(). 00 1 . This would be considered a small (0.291) Cohen’s d (1992) effect size. A 
significant difference (increase) on the academic participation PSS factor pre (M= 2.53, 573=1.02) and post 
(M= 2.81, 573=. 942), ((80)=-2.84, /?=(). 006 was also identified. This would also be considered a small (0.285) 
Cohen’s d ( 1992) effect size. 

Treatment group analysis of (-test pre-post data (for students who attended the field trip and 
met/interviewed scientists) revealed a significant difference (increase) on the academic desire PSS factor 
pre (M= 2.59, 573=. 870) and post (M= 3.28, 577=1.16), ((58)=-5.246, p<0.00 1 . This would be considered a 
large (0.668) Cohen’s d (1992) effect size. Also, a significant difference (increase) on the academic 
participation PSS factor pre (M= 2.50, 577=. 771) and post (47=3.03, 577=1.13), ((62)=-2.84, /?<0.00l was also 
identified. This would be considered a medium (0.553) Cohen’s d ( 1992) effect size. 

4.2 Experiment and Control Groups at Post-test 

Analysis of variance, one-way between subjects ANOVA for post-test data, was conducted to compare the 
effect of meeting and interviewing scientists on PSS desire and participation. There was a significant effect 
as a result of the treatment on PSS desire at the p<. 05 level for the two conditions [/*'( 1 , 1 38 )=5 .53 1 , 
p=0.020)] treatment and control. The mean score for the treatment condition (M= 3.239, 5/3= 1 . 1 72) was 
significantly different (higher) than the control condition (A7=2.787, 573=1.045). This would be considered a 
small-to-medium (0.404) Cohen’s d (1992) effect size. There was not a significant effect as a result of the 
treatment on the PSS participation factor at the p<. 05 level for the two conditions [l 7 ( 1 , 1 45 )=2. 1 35, 
p=0.160)]: treatment and control. 


5. DISCUSSION 

Student preferences, as the effect of liking science-related content, and student learning, as cognition, are 
separate yet related (Zajonc, 1980) aspects of students’ attitude towards content areas such as science or 
STEM. These aspects are both elements of informal learning. Recognizing the importance of informal 
leaning to student attitudes towards STEM, this study examined participation in STEM activities that provide 
students with opportunities to interact with science-related content and to develop a liking for STEM in the 
real world — outside the classroom context. Positive science experiences, such as science field trips and 
exposure to interesting persons of science, will support positive attitudes towards science (Aschbacher et al., 
2010; Pell & Jarvis, 2001 ). Preliminary findings from this study indicate that students’ perceptions of being a 
person of science and their interest in doing science became more positive after a science field trip 
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experience. This confirmation of students’ more positive perceptions of being a person of science (one who 
will participate in science) as the result of an out-of-school science experience emphasizes the importance of 
providing students with opportunities to have these science-related experiences and also to feel supported in 
science interests in multiple communities (Schneider & Stevenson, 1999; Aschbacher et al., 2010). 
Additionally, results of the experiment showed that students who met and interviewed scientists in small 
groups were significantly more positive in their desire to become a scientist than students who did not have 
small group interaction with scientists — emphasizing the value of giving students the opportunity to see and 
meet role models (Smith & Erb, 1986). 

Students will naturally aspire to imitate persons who are considered important, such as heroes and 
celebrities (Pleiss & Feldhusen, 1995; Sjpberg, 2002). However, there is some question as to whether or not 
scientists are considered to be important and interesting role models. In order to understand how students 
view STEM professionals, it is important to understand students’ stereotypes of scientists. The stereotypes 
that students have of scientists have been previously examined through draw a scientist studies in order to 
allow an understanding of how people, especially children, picture scientists. Findings from this study 
support the notion that students who have the opportunity to become familiar with people of science (Smith 
& Erb, 1986) and have positive support for science -related interests by scientific role models whom they 
view as being interesting will have more positive science -related attitudes (Aschbacher et al., 2010). In 
addition to examining students’ science self-concept after meeting STEM role models, participants in this 
study also drew a scientist. Study participants were found to be more positive regarding their science 
identities. Additional analysis is planned to compare the drawings of these students to other student groups in 
order to see if study participants differ in stereotypic views of STEM professionals from other student 
groups. The authors suspect that students who have the opportunity to meet scientists, as did the participants 
from this study, may tend to have less stereotypic depictions of STEM professionals. Additional research 
conducted by these authors will include the analysis of possible relationships between students’ views of 
scientists and their own scientific identities. Ongoing research on factors that support science identity for 
participation in science is also suggested. The authors recognize that it will be difficult to gauge the 
long-term impact of informal learning activities, such as meeting scientists, being supported in science 
interests, and experiencing science -related field trips, will have on students. However, there is evidence to 
suggest that experiences that support a strong, positive science identity are not only related to younger 
students’ interest and participation in science but are also related to persistence and success in attaining 
science-related academic goals as a part of students’ higher education. 
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ABSTRACT 

During the last few years, ongoing developments in the technological field of Cloud computing have initiated discourse 
on the potential of the Cloud to be systematically exploited in educational contexts. Research interest has been stimulated 
by a range of advantages of Cloud technologies (e.g. adaptability, flexibility, scalability, accessibility, cost effectiveness). 
For these reasons, there have already been efforts concerned with the educational uptake of the Cloud at different scale 
levels (mainly at the institutional and regional level). Apart from that, there are also larger-scale initiatives taking the 
form of research projects (e.g. the Rural School Cloud project). Nevertheless, most practices have been mainly shaped by 
the application of economic policies rather than systematic research. In this emerging situation, the aim of this paper is to 
present the activities of a European network, which has been set up to research how education should adapt and respond 
to advancements in Cloud computing. The innovative character of the network is that it attempts a holistic approach to 
the investigation of the effects of Cloud technologies on formal education by targeting at all involved stakeholders, and 
contexts of use, at all educational levels. By analytically presenting the adopted methodology and network members’ 
vision for Cloud-based, student-centered learning and teaching, our aim is to provide a detailed account of the network’s 
research activities. 


KEYWORDS 

Cloud computing. Cloud-based education, European network, student-centered learning. 


1. INTRODUCTION 

During the last few years rapid technological advancements have taken place with unprecedented effects on 
the ways in which people communicate and socialize, get informed, work, conduct their everyday exchanges, 
and, of course, learn. Technological developments and their impact on everyday lives and habits of people in 
general, and young people particularly, have major implications for decision-making and strategic planning 
on the quality and nature of educational services offered by formal education establishments. Personalization 
of learning is at the core of challenges that formal education faces with the focus being on supporting learners 
develop and carry out learning plans at their own pace. In this context, Saveri (2013) proposes the “ 
create-your-own-school ” movement as a future educational trend. With the use of this term, specific 
emphasis is posed on the need for schools to adapt to students’ preferences, interests, and learning needs, as 
well as to their surroundings and particularities of communities in which they belong. Similarly, the NMC 
Horizon Report (Johnson et al, 2015) lists the broad adoption of a blended-learning approach, by higher 
education institutions, as a key, short-term challenge. In this way, educational organizations will be able to 
promote and facilitate instruction and learning beyond the confines of traditional classrooms and rigid time 
schedules, at each student’s convenience. On the other hand, financial crisis has led to significant reductions 
in public expenditures on education. Despite the increasing demand for education and training, austerity puts 
significant limits in the funding of educational organizations, which have to “Jo more with less in the coming 
years” (OECD, 2013). Moreover, due to fast-paced technological changes, the need to maintain infrastructure 
and keep it up to date entails costs that may not be within an educational institution’s budget (Sultan, 2010). 
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In this emerging landscape. Cloud computing promises to improve the efficiency of organizations and 
actors involved in education and optimize the technology-enhanced learning experience in an affordable way. 
Cloud computing is defined as “a model for enabling ubiquitous, convenient, on-demand network access to a 
shared pool of configurable computing resources (e.g. networks, servers, storage, applications, and services) 
that can be rapidly provisioned and released with minimal management effort or service provider 
interactions ” (Mell & Grance, 2011). According to the proposed definition, the Cloud is composed of “five 
essential characteristics ” (namely, on-demand self-service, broad network access, resource pooling, rapid 
elasticity, and measured service), “ three service models ” (namely, Software as a Service - SaaS, Platform as 
a Service - PaaS, and Infrastructure as a Service - IaaS), and “four deployment models ” (namely, private 
Cloud, community Cloud, public Cloud, and hybrid Cloud). Cloud technologies have the potential to improve 
the quality of provided educational services by allowing: (i) access to anything from anywhere in anytime on 
any device by anyone, (ii) sharing of knowledge that facilitates collaboration, (iii) remote interactions and 
monitoring of learners’ progress, (iv) processing of large data sets through the availability of increased 
computational power, and (v) access to state-of-the-art tools and services without the need of purchasing and 
maintaining hardware or installing and updating software (Sultan, 2010). 

Given all the above, the aim of this paper is to provide a detailed account of the activities of a European 
network, namely the School on the Cloud network (http://www.schoolonthecloud.eu/), which has been set up 
to investigate how formal education should adapt and respond to Cloud computing developments and whose 
activities are funded by the European Commission under the Lifelong Learning Programme (Information and 
Communication Technologies Action - Key Activity 3). More specifically, our intention is to focus on issues 
regarding the methodology through which network members research the potential of the Cloud to facilitate 
student-centered, personalized learning. To this end. after reviewing existing Cloud-based education efforts 
and initiatives, we proceed with the description of: (i) the network’s structure and operation, (ii) findings of a 
survey among network members, and (iii) the context in which personalization of learning on the Cloud is 
envisioned. 


2. EXISTING CLOUD-BASED EDUCATION PRACTICES 

There are a number of Cloud-based education implementations and initiatives that have taken place in 
various contexts, and at different scale levels (Donert & Bonanou, 2014). Sultan (2010) provides accounts of 
Cloud-based solutions that have been embraced by educational organizations with higher education holding a 
greater proportion in Cloud use cases. Lor instance, the University of Westminster has turned to the adoption 
of a Cloud-based e-mailing service, because of the scarce use of the institutionally hosted e-mail system, as 
well as the provision of a suite of communication and productivity tools (namely instant messaging, shared 
calendar, word processor, spreadsheets and presentations applications) able to support collaborative learning 
activities. Budget requirements for the implementation of the recruited solution were low and consequently, 
provision of quality services in a cost effective way became feasible. Apart from that, another advantage of 
Cloud-based services is the availability of computational power for processing large data sets in an affordable 
way. This was the rationale behind the strategic decision of the Medical Center of Wisconsin Biotechnology 
and Bioengineering Center, in Milwaukee, to rent processing time on Cloud servers. As a result, analysis of 
research data became less expensive, whereas sets of raw experimental data along with results of conducted 
analyses become accessible to the wider research community. 

Cloud-based education initiatives have also taken place in school education contexts. An indicative case 
is that of Kentucky’s Pike County district, where Cloud computing solutions were adopted by its schools in 
an attempt to reduce public education expenditures. Among others, schools benefited from the transformation 
of a large number of old computers into fully functional virtual machines. With no need for purchasing and 
upgrading hardware, as well as keeping installed software up to date, available computers could be reused by 
receiving processing power and software from Cloud servers. As a result, teachers and students were able to 
become more productive by focusing on their tasks and activities without any concerns about infrastructure 
maintenance issues. Another example is that of Brescia House School (in South Africa) presented in the 
official website of Microsoft (https://customers.microsoft.com/Pages/CustomerStory.aspx?recid=300). By 
substituting the previously used LMS for a more usable and flexible platform, hosted by Microsoft, school 
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authorities managed to provide teachers and students with device-independent functionalities for accessing 
and sharing resources, e-mail services, personal storage spaces, as well as the potential to communicate and 
collaborate even through the use of social media. Apart from the above, there are large-scale efforts that have 
been concerned with the investigation of the Cloud’s educational potential, such as the Rural School Cloud 
project (http://rsc-project.eu/). This project targets at the examination of methods of providing optimized IT 
solutions to schools that are located at rural areas, or away from cities, with heterogeneous populations. 
Based on EU-driven strategies that emphasize the importance of enabling schools with special needs (e.g. 
schools in rural areas) to access networked digital technologies, the Rural School Cloud project seeks to 
explore how Cloud technologies may offer learning solutions to highly diversified student populations. 

All the above described cases of introducing Cloud-based technologies in formal education, or examining 
the educational potential of the Cloud, are different in terms of scale levels and incentives. Except for the 
Rural School Cloud project, most Cloud-based education initiatives have taken place with the aim to address 
issues faced at the local (e.g. institutional) level. Larger-scale implementations (e.g. at the regional level) 
have indeed been based on policies which, however, have been mainly formatted by economic reasons and 
not by systematic research. The innovative character of the School on the Cloud network’s research is that it 
attempts a holistic approach to the investigation of the effects of Cloud computing technologies on formal 
education. By targeting at all involved stakeholders and contexts of use, as well as all levels of education, and 
by focusing on the individual learner, the network’s intention is to raise awareness about potential benefits, 
provide guidelines and inform, and stimulate further research. 


3. DESCRIPTION OF THE SCHOOL ON THE CLOUD NETWORK 


3.1 Definition of the Research Problem 

The School on the Cloud network targets at investigating how formal education should respond to fast-paced 
technological advancements and especially to the proliferation of Cloud technologies. More specifically, the 
intention is to narrow the existing divide between Cloud computing developments and their uptake by formal 
education establishments. To this end, the following two research questions are attempted to be addressed: 

• How should formal education establishments respond to Cloud Computing developments? 

• What is the impact of available Cloud Computing technologies on exercising educational policy, teaching 
practices, and on facilitating individualized learning processes? 

3.2 Methodology 

For the purpose of addressing the above stated research questions, a European network (namely, the School 
on the Cloud network), in which all educational sectors and levels are represented and almost all categories 
of educational stakeholders are included, has been set up to undertake a three-year research project (January, 
2014 to December, 2016). The School on the Cloud network, whose development is in alignment with the 
Lifelong Learning Programme’s ICT - Key Activity 3, consists of 57 partners including 21 Universities and 
teacher training departments, 9 NGOs, 8 schools, SMEs, research institutes, adult education and VET 
providers, a European professional association, and a library. Tasks of the network are: (i) the research of 
“state-of-the-art”, (ii) the production of a series of core publications on the exploitation of Cloud in different 
educational contexts, (iii) the production of guidance resources, (iv) the establishment of methods and means 
for network members to share their findings and expertise, (v) the dissemination of research outputs within 
and beyond the confines of the network, and (vi) the establishment of links with other European-wide or 
nation-wide initiatives. 

Research by adopting different points of view (i.e. the educational organization administrator’s, teacher’s, 
and learner’s point of view) is achieved through the assignment of network partners to four Working Groups 
(WGs) according to their fields of expertise and interests. These Working Groups are: (i) the innovative 
Manager (i-Manager) WG, (ii) the innovative Teacher (i-Teacher) WG, (iii) the innovative Learner 
(i-Learner) WG, and (iv) the innovative Future (i-Future) WG. In order to optimize the project’s management 
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and better monitor progress, a leader has been assigned to each WG. WG leaders along with the project 
coordinator form the project’s Steering Group. Table 1 below contains descriptions of each Working Group 
and their tasks in terms of assigned Work Packages (WPs) and associated deliverables. Each deliverable of 
each WP constitutes part of one of a four-phase workflow that dictates a sequence for the implementation of 
each WG’s tasks. These phases are: (i) awareness raising/motivation, (ii) review of literature/best practices/ 
available tools, (iii) development of guidelines, and (iv) dissemination of results and transfer of expertise. 
Deliverables marked with an asterisk have already been developed and are available to the public through the 
project’s official website and dedicated social media websites. 

Finally, given the number of participating institutions, an issue of concern is the coordination of members 
of the network and their collective activities. To this end, a number of events, planned to take place at regular 
time intervals, have been scheduled. These events are: (i) network summits , which are open to the public and 
take place on an annual basis with the participation of all network members, (ii) WG meetings, in the context 
of which members of each WG share outcomes of ongoing, or completed, activities and monitor progress 
against agreed upon time schedules, and (iii) Steering Group meetings, conducted with the participation of 
WG leaders and the project coordinator with the aim to evaluate progress and prioritize next activities. Figure 
1 below illustrates key tasks relevant to the project’s development. 

Table 1. Description of the School on the Cloud network’s Working Groups and their project -related tasks 


WG name and description 

WG1 (i-Manager): Examines aspects of 
educational leadership, management and 
organizational change. Aims to identify and share 
technological, social, economic, cultural and 
other experiences in different educational 
contexts, as well as provide guidance to 
educational organizations. 


Assigned Work 
Package (WP) 


WP2: Leading/ 
managing 
Cloud-based 
developments 


WG2 (i-Teacher): Explores the impact of the 

Cloud on the roles of teachers and trainers and 

examines how Cloud-based developments can 

find their way in education. It attempts to identify WP3: i-Teacher: 

barriers and required competences, review innovative-Teacher 

learning and teaching approaches, and provide 

practical and essential guidance for teachers and 

teacher trainers. 


WG3 (i-Learner): Brings together teachers and 
educators, schools, colleges, and adult education 
organizations with the aim to exploit the 
opportunities resulting from both formal and 
informal learning situations. WG3 will define 
personalized learning, and from existing best- 
practice case studies, will develop a guide on 
how to facilitate it. 

WG4 (i-Future): Deals with topics like the role 
and the impact of open (education) resources 
through the Cloud, new generation Cloud-based 
tools, and issues such as ethics. Efforts aim to 
attract the interest of organizations such as 
education and teacher associations, NGOs, and 
museums, as well as researchers, educators, 
school administrators and policy makers. 


WP4: Integrating the 
Cloud: personalized 
learning 


WP5: Future 
scenarios for 
Education on the 
Cloud 


Associated deliverables 

D2.1. Education and the Cloud - a summary 
leaflet for leaders and managers (*) 

D2.2. Publication: policy, leadership and 
management issues 

D2.3. Implementation guidelines and advice 
D2.4. Workshop 

D3.1. Presentations on the impact of 
Cloud-based teaching and teacher education 
on teachers and trainers (*) 

D3.2. An online catalogue of platforms, 
tools and apps for teachers, trainers and 
educators (*) 

D3.3. A guidance leaflet on Cloud teaching 

and Cloud learning 

D3.4. Workshop on teachers’ needs 


D4.1. State of the art literature review of 
personalized learning and the Cloud (*) 
D4.2. Case studies of personalized learning 
D4.3. Guidance on personalized learning 
D4.4. Workshop on personalized learning 


D5.1. A review of Cloud-based futures and 
methodologies (*) 

D5.2. Workshop on “Futures: thinking on 
the Cloud” 

D5.3. Cloud-based education: scenarios for 
the future 

D5.4. “The Future of Education on the 
Cloud” 
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2014 


2015 


2016 


2017 


STAGE 4: INCREASING INFLUENCE 
contribution to reform 
professional development and support 
future perspectives 
shared values 



website leaflet 


state-of-the-art 
guidance platform 


Work Groups’ 
publications 


DISSEMINATION 


Figure 1. Tasks related to the adopted by the network methodology 


3.3 User Needs Analysis: Perceptions of the Cloud and its Educational 
Potential 


A user needs analysis was conducted prior to the first network summit (took place in March, 2014) with the 
aim to investigate perceptions of the Cloud and its characteristics, perceived educational affordances, beliefs 
about potential impact of the use of Cloud technologies on the roles of teachers and trainers, and reasons that 
may hamper uptake by organizations. This formal process also facilitated the reflection of network members 
on their expectations and offered a starting point for all Working Groups’ research by providing input to 
defined tasks. User needs analysis was implemented through an online survey and with the participation of a 
sample of N=59 representatives of all network members. 


Virtualization of computers 


Sharing of computer space 


Services offered online 


Hosted Internet space 

Computers connected 
through the Internet 

Allocation of space to users 
on -demand 




Figure 2. Perceived definitions of the Cloud Figure 3. Cloud characteristics enabling educational exploitation 

As far as perceived definitions of the Cloud are concerned (“ Which of the following applications does 
Cloud computing mean to you?”), 35.6% of participants replied that the Cloud relates to hosted Internet 
space, 17.9% replied that it relates to services offered online, and 13.6% of responses offered views of the 
Cloud as computers connected through the Internet or virtualization of computers. Responses to this question 
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are illustrated in Figure 2 above. With respect to characteristics that may facilitate use of Cloud technologies 
in educational contexts (“Which of these characteristics does the Cloud offer for education?”), the most 
frequently cited responses were “ increased capacity” (23.7%), “independence of location” (20.3%), “reduced 
costs” (15.3%), and “more capabilities” (13.6%). An analytical presentation of results is offered in Figure 3 
above. Additionally, from responses to the question “ From your perspective, what makes using the Cloud in 
education innovative ?” the following list of issues arose: (i) freedom from hardware and disc storage, (ii) use 
in all forms of learning situation, (iii) potential for learning beyond traditional classroom situations, (iv), 
many innovative apps and tools, and (v) open access - MOOCs. 

A key target of the user needs analysis process (having direct implications for research undertaken by the 
i-Teacher WG) was the investigation of perceptions of the impact that Cloud-based education may have on 
the roles of teachers and trainers (“What impact does Cloud-based education have on the roles of teachers 
and trainers in education?”). Data gathered from participants’ responses reveal that a shift towards a 
Cloud-based education paradigm implies that teachers need to adopt the role of a catalyst for change, 
mentor/coach, co-learner, information manager, and facilitator of the learning process. Apart from that, a 
focus on issues impeding the uptake of Cloud-based solutions by institutions could not be left outside of the 
survey’s scope. To this end, respondents were asked to rate, on a 0 to 9 scale, a number of potential reasons 
made available as part of a specifically-defined question (namely, “On a scale of 0-9, consider the relative 
importance of these possible issues in influencing the take up of the Cloud for education in your 
organization.”). Reasons that were rated the most, were: (i) security (6.47), (ii) culture of use (6.35), (iii) 
infrastructure and trust in the system (6.15), and (iv) capacity and management (6.13). Figure 4 below 
presents the full range of responses. 

Knowledge 
kfanagement 
Capacity 
Trust in the system 
Infrastructure 
Culture of use 
Security 

ISO 6,00 6,20 6,40 6,60 4,60 5,10 5,60 6,10 

Figure 4. Issues that may hamper adoption of a Cloud-based paradigm by formal education establishments 

A cursory review of survey findings reveals that perceived Cloud definitions and characteristics, as well 
as reported impacts and reasons affecting the Cloud’s educational uptake, are in line with issues presented in 
literature. This conclusion has provided a good starting point for the network’s efforts with ongoing research 
having the potential to lead to multiscale, sustainable Cloud-based educational solutions. 

3.4 Focus on the Learner: Facilitating Cloud-based Personalized Learning 

It is generally admitted that we cannot envision the future of education (on the Cloud) unless a paradigm shift 
(i.e. a fundamental change in methods of delivering education) takes place. In simpler terms, full exploitation 
of the Cloud in education may only be achieved through a paradigm shift from the teacher -centered approach 
to Cloud -based, student-centered teaching and learning (Koutsopoulos & Kotsanis, 2014). In this context, the 
aim of this section is to present the School on the Cloud network’s vision for a Cloud-based, student-centered 
model of learning by attempting to draw links between affordances of Cloud technologies and contexts of use 
focused on the individual learner and specified by existing instructional strategies. 

To begin with, services and tools made available through the Cloud can be utilized in support of direct 
instruction approaches. More specifically, according to Gonzalez-Martinez et al (2015), Cloud-based 
video-on-demand services can be exploited by educational organizations with the aim to deliver lectures to 
their students. Video-recorded lectures may benefit learners by allowing them to interact with educational 
content anytime, anyplace, at their convenience. By having immediate access to video-recorded educational 
material, learners are able to go through presented concepts as many times as they need and thus, develop 
knowledge at their own pace. Apart from that, provision of increased computational power through the Cloud 
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High costs 
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offers the potential for interactions with demanding, in terms of required hardware resources, learning 
environments, such as simulations (Gonzalez-Martinez et al, 2015). With the processing taking place in the 
Cloud, learners can get involved in simulation -based learning scenarios where learning occurs in a 
meaningful way through interactions with the medium. 

Cloud technologies provide affordances for learning that not only fits the behaviorist learning paradigm, 
but their main advantage is that they are capable of facilitating learning in social contexts. As Denton (2012) 
points out, functionalities of Cloud-based applications and tools that support and promote collaboration and 
communication allow learners to become involved in joint learning activities. For instance. Cloud service 
providers offer online suites of tools with collaborative features, able to be accessed simultaneously by many 
users, which can significantly enhance the experience of remote interactions. Besides, according to the 
socio-cultural learning theory, learning at the individual level involves making meaning as part of the 
learner’s engagement in appropriately structured, collective processes. Social interaction is an “ essential 
component ” of the problem-based learning approach (Eggen & Kauchak, 2006, p. 252) and can significantly 
enhance the benefits of involvement in project-based learning (Boss & Krauss, 2007, p. 65). Hence, 
incorporating Cloud-based applications and tools into appropriately designed problem-based and 
project-based learning scenarios facilitates learning in a way that is meaningful to the learner and relevant to 
real-world contexts. 

By taking advantage of Cloud technologies there is also potential to maximize benefits from the delivery 
of inquiry-based learning scenarios. More specifically, lower-level Cloud services, such as storage spaces in 
the Cloud, can be used by learners ( and teachers) to store, share and immediately access sets of data produced 
from their involvement in scientific inquiry (Abrams, 2012). By this way, learning activities are not confined 
to the school/university science laboratory and are not limited by strict time schedules. Additionally, through 
the provision of Cloud-based suites of productivity tools learners can engage in data elaboration processes, 
individually or in collaboration with peers, and present their findings. Elaboration of scientific data resides at 
the core of inquiry-based learning (Joyce, Weil, & Calhoun, 2009, pp. 159-187) and thus, providing learners 
with the appropriate Cloud-based tools and applications helps them to focus on inquiry-based activities rather 
than being bothered with issues of practical and technical nature. 

However, the contribution of Cloud technologies to the personalization of learning can also be considered 
in terms of the means provided to teachers for the delivery of quality learning experiences. In such a context, 
the Cloud’s potential lies at the emergence of Cloud-based Learning Object Repositories (LORs), able to 
balance supply and demand, improve availability of resources, respond better to queries for learning objects, 
and facilitate interoperability (De la Prieta et al, 2014). Furthermore, the existence of providers of Cloud 
services specializing in storing and managing of specific types of digital resources (JISC, 2012) may lead to 
new, decentralized LOR models enabling the storage of different types of learning objects to fit-for-purpose 
spaces. Developments of this kind offer teachers the opportunity to access quality material, either stand-alone 
or as part of integrated modules and courses, anywhere, anytime, according to their needs (Silva & Donert, 
2015). Thus, teachers can create, edit and share innovative learning scenarios, with the use of media-rich 
content, by taking advantage of quality Cloud-based services and tools. 


4. DISCUSSION AND IMPLICATIONS FOR FURTHER RESEARCH 

During the last few years, educational organizations have begun to embrace Cloud-based solutions mostly in 
their attempts to reduce expenditures (Sclater, 2010; Sultan, 2010). However, the Cloud’s actual potential lies 
at a number of affordances able to facilitate learning in many different contexts and based on a broad range 
of instructional strategies. For instance. Cloud technologies offer opportunities for individualized learning by 
allowing for anytime and anyplace access to media-rich learning resources, state-of-the-art tools and virtual 
learning environments. Furthermore, communication and collaboration functionalities of Cloud-based tools 
and applications help learners to make meaning at the individual level as part of their active participation in 
collaborative learning activities. 

In this emerging landscape, the School on the Cloud network has been formed to explore how education 
should respond to new IT developments. Through a rigorous research methodology, the network’s activities 
target at a systematic review of “state-of-the-art”, considered both in terms of available Cloud technologies 
and existing Cloud-based educational initiatives, and the provision of guidance and support to stakeholders. 
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However, full exploitation of the Cloud in education requires careful reconsideration of the roles of teachers 
and learners. Therefore, by focusing on the individual learner, and with the help of well-established processes 
of expertise sharing among network members, the intention is to develop a body of knowledge regarding the 
impact of the educational use of the Cloud, as well as potential issues of concern. Research activities aim to 
result, among others, in publications providing incentives and guidelines for broad uptake of the Cloud and 
proposing sustainable solutions at various scale levels. There is much to do for bridging the gap between 
current practices and the School on the Cloud network’s vision for Cloud-based, student-centered learning. 
To this end, the project consortium pursues dissemination of knowledge and expertise, by establishing links 
with other research initiatives, as well as stimulation of further research on a number of issues, such as: (i) 
evaluation of future scenarios of education on the Cloud, (ii) design, application, and evaluation of 
context-specific Cloud-based educational solutions that span across all levels of education, and (iii) 
development and testing of Cloud-based services and tools able to cater for special educational needs (e.g. 
assessment). 
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ABSTRACT 

The flipped classroom approach (FC) is for the educational world a chance of recovery and improvement of pedagogical 
student -centered model and collaborative teaching methods aimed at optimizing the time resource and to promote 
personalization and self-learning in a perspective of autonomy. 

The paper moving from a pedagogical reflection on innovative methodology for the improvement of teaching in schools 
and universities, presents the model of Flipped Classroom activated during 2014-2015 at the University of Salento in the 
experimentation of the E -Learning in the University by focusing on the pedagogical model implemented: theoretical 
framework, research objectives, development phases and testing 


KEYWORDS 

Flipped Classroom, technologies, university, pedagogical student -centered model 


1. TEACHING SKILLS AND FLIPPED CLASSROOM APPROACH 

Today schools and universities face challenges and demands more and more articulated that reflect the 
complexity of the systemic and social model which try to respond, according to the paradigm of 
constructivism, to new instances from the world of production and the multiple needs of new generations of 
students. 

In the conclusions of the work of Lisbon the European Parliament, in 2000, indicated some ways to renew 
teaching, considering that transmissive teaching often generate in students demotivation, alienation and 
disaffection for study (Da Re, 2013). In the Recommendations of 18 December 2006, are set out in a 
definitive way the eight key competences for European citizenship by defining the concept of competence as 
"a combination of knowledge, skills and attitudes appropriate to the context [...] those which all individuals 
need for personal fulfilment and personal development, active citizenship, social inclusion and employment". 
Teachers are asked to promote students learning through experience and through induction processes, 
whether by encouragement for knowledge representation. 

According to many studies Italian university students have to be structurally considered as digital natives 
(Cavalli et al., 2009). Considering that, in order to effectively offer students opportunities useful to build the 
expertise it is necessary to provide tools, techniques and strategies centred on competence. 

Alongside the lessons it is necessary to provide discussions, group work, case studies, solutions of 
problems of experience, taking of decisions, realization of meaningful tasks, because the learning motivation 
is the result of two conditions: perceive to be able to tackle the task and feel that the effort required has a 
value and meaning (Brophy, 2003). 

This structure often not combined with a limited time in the classroom or the limit of credits assigned to 
the discipline, but digital technology can be a valuable ally. 

We know a lot about the phenomenology of technological innovation, that are generated at each wave 
peak expectations, accompanied by recurrent mythologies, were followed by failures and advent of new 
waves (Oppenheimer, 2003; Ranieri, 2011). 
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Asked if there is a positive relationship between technology and learning the answer is negative: often we 
find that the use of new technologies in school is not in itself effective; consequently are the methods and not 
the technology itself that make a difference in learning outcomes (Clark et al., 2006; Hattie, 2009), 
however, among the technologies the main benefit is presented in the use of interactive video (Calvani, 
2012 ). 

Looking at the contribution of teaching methods, the cooperative approach and the laboratory didactics 
are those that the scientific literature and best practices attest as functional and productive. It is educational 
settings designed to make students think and act on deliveries active. 

Recent education research is focused on how teachers can improve their didactics and use the class-time 
more efficiently than the traditional lesson. A way to create a didactic more interesting through the use of 
technology is the flipped classroom or flipped learning. 

The flipped classroom structure gives students the opportunity to practice in-class what they are learning, 
which is consistent with the constructive alignment approach recommended by Biggs and Tang (2007). 

The flipped classroom structure demands active engagement both from the students and teachers. Sam 
and Bergmann suggest that teachers "flip" their class to utilize the time most effectively. They propose that 
that students, prior to attending class should read a chapter, watch a video or explore a new topic. Then, the 
teacher may facilitate a discussion based on this information to deepen the students' understanding. Looking 
at the contribution of the cooperative approach teaching methods and teaching laboratory are those that 
literature and best practices attest as functional and productive. 

An educational structure able to recover time and experiential workshop classroom to dedicate it to the 
activation of cooperative tasks, peer learning, workshops and educational problems, is given by Flipped 
classroom (class upside down). 

The FC approach refers to an inverted teaching, ie a model which locates in face to face teaching not the 
transmission of knowledge, but rather the deepening and sedimentation of learning through exercises and 
critical reflections downline of an individual study, but structured that the student performs independently 
choosing time and space for learning according, however, to time ranges and specific deliveries proposed by 
teachers. The study, therefore, is not a proxy by the teachers, but a chance for students to personalize and 
self-regulate his learning process (Franchini, 2014), while teaching, traditionally represented by the 
combination of explanation-homeworks, becomes a process in several stages aimed on the one hand to 
accompany the learning through the promotion and realization of learning materials prepared ad hoc 
(video-classes, network resources, books etc), the other hand a moment to dispel concerns through dialectical 
discussions and to support the learning through exercises and groupal or individual in-depth analysis (think 
of the teaching laboratory and cooperative learning) by promoting to students thinking (Khan, 2012) and 
self-management (Fulton, 2012) skills increasingly complex. 

Therefore the entire educational process to undergo a real inversion in response - these are the premises 
from which the promoters are started - to the needs of the teachers to optimize and not waste time in the 
classroom. The teachers, indeed, are increasingly gripped by school programs endless that steal their time 
limiting the moments of study and exercise to isolated and episodic experiences. 

The FC approach is not a pedagogical model, it doesn’t have an epistemology well-defined: it is the result 
of a multitude of experiments and best practices empirically poorly controlled and comparable to each other, 
created to satisfy the needs that come from the world of education and aimed recovery pedagogical model 
learner-centered aimed to customization and sharing of learning (Bloom, Vygotsky or teaching methods as 
Peer-Assisted, Tutoring, Collaborative and cooperative learning) according to a perspective of optimization 
of the school time and empowerment the autonomy of the student. 

This autonomy is not only about the approach to the study, but it affects the democratization of 
knowledge and, therefore, the sphere of student participation in the co-construction and sharing of knowledge 

Franqueira and Tunnicliffe (2015) performed an exploratory study on this topic following the Critical 
Interpretive Synthesis methodology for analysis of the literature. 

According to them words like flipped learning and flipped classroom are often used interchangeably as an 
indication of innovation, flexibility, creativity and pedagogical evolution and they indicated that the term 
“Flipped Learning” is misleading and that, in fact, the synthetic concept behind it is “Flipped Teaching”. 
They realized a synthesising argument, in the format of two synthesis models, of the potential benefits 
promoted by flipped teaching and the potential issues which affect its success in practice. 
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2. FLIPPED LEARNING IN HIGHER EDUCATION 

Despite the FC appears more prevalent in the context of school, the world of higher education promotes more 
and more experiences of research and teaching that refer to this approach. 

In fact, in recent years in academic and lifelong learning have been launched several experiments aimed 
at guide the student in the process of acquisition of knowledge and skills of disciplinary knowledge highly 
codified (mathematics, physics, chemistry, etc.) in which the risk of failure of learning and dropouts is very 
high. 

In that sense, if in school this approach responds to a need to rationalize and optimize the limited time 
available, in the academic context, characterized by a high degree of freedom and management (which does 
not mean autonomy) of time and learning, this approach performs a specular function: return to students, 
through precise training deliveries, assigned by the teachers, the "weight" and the "sense" of learning time, 
orienting them, then, to a self and competent managing of it. 

Such operation, if in one hand is intended to contain the wasting of resource time and, consequently, to 
empower students, on the other hand gives them the ability to customize their own learning process 
according to their educational needs (styles earning, special needs, etc.) and life needs (business and familiar 
needs). 

In response to these needs, the FC approach uses structured learning contents and disseminated in 
asynchronous mode that become preparatory to the recovery of the content in the classroom. Most 
educational experiences FC refers to an educational e-learning or blended learning and uses video lessons 
and learning materials available and shared across the internet. 

There is an extensive literature on the attitudes and perceptions of students about the use of video lessons 
in education (Bolliger et al., 2010; Fernandez et al., 2009; Hill & Nelson, 2011; Lonn & Teasley, 2009; 
Chester et al., 2011), while a scientific debate about FC approach is lacking. Bishop and Verleger (2013), 
indeed, have examined 24 empirical studies on this issue highlighting that a strong methodological and 
content heterogeneity often are not supported by empirical data. 

In contrast, however, in recent years on the web it is observed a proliferation of blogs, websites and 
videos aimed at the promotion and sharing of FC teaching (like Flipped Learning Network). 


3. TIC & DIL PROJECT: FRAMEWORK AND DESIGN 

The Tic & DIL project: Information Technology and Communication and Teaching of Reading, has been 
developed by the working group of the Center on New Technologies for Inclusion of the Dpt. of Fhstory, 
Society and Fluman Studies within a PON for the development of E-learning in the University. 

The flipped classroom approach used in this study was undertaken in Semester 2 (February- June 2015). 

The project aimed to develop a learning environment/workshop for students of the undergraduate and 
graduate program of the Faculty of Education. 

The course was organized in two interdisciplinary thematic unit (ITU) implemented according to the 
flipped classroom model. 

Common theme of the two ITU was: LANGUAGES and READING, i.e. insights and workshops aimed 
to promoting in the university context interdisciplinary links about learning languages mediated by 
technology of dyslexic student (fig 1 ). 

The ITU bound bachelor students has involved teaching and teachers of literary theories and methods of 
education and teaching methods Laboratory. 

The ITU intended for students of degree involved teachings and teachers of the Laboratory of educational 
planning and of theory and techniques of observation of behaviour in education. 
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Figure I . Interdisciplinary thematic unit (ITU) 

The online learning has been divided into multimedia and thematic lectures designed with an hypertextual 
logic with videos and simulations in order to encourage the interaction among students. 

For the construction of educational activities the students have been used proprietary and open source 
software (Camtasia, eXeLearning, Edpuzzle, Storyline, Xerte, Prezi) that have allowed to reuse video assets 
available on major web portals. 

The project and teaching activities have lasted 5 months (February- June 2015) and were designed to 
recognize and evaluate: the learning outcomes achieved by students in both groups (see the experimental 
model); the effectiveness of the FC teaching model; the critical elements about technological choices 
adopted. 


4. EXPERIMENTAL MODEL 

The project was implemented through the use of the Moodle platform of the University 
(http://formazioneonline.unisalento.it/) (Fig. 2). 



Figure 2 . Entry page in Moodle and video lesson example 

The investigation involved a total of 380 students. The experimental design included a control group 
(CG) and an experimental group (EG): the CG included those who voluntarily entered to the experimental 
model (260 between graduate and undergraduate students); CG (120 students) included those who had 
chosen to follow the traditional teaching model. 

Inside the EG there was activated a second level of testing. A portion of the EG (25 students), followed a 
blended learning, so that for some activities, the contents that would have been the subject of subsequent 
lessons were anticipated with handouts or videos online. 
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4.1 Planning and Time 

The two groups attended a similar program in terms of content from March until late April, based on a 
traditional teaching (lectures) and workshop activities in the classroom. The EG used the Moodle platform in 
order to support teaching (access to learning materials used in class, handouts and maps of synthesis) and in 
support of the interaction (personal messages and group activities for the survey data with questionnaires; 
intermediate deliveries of papers and projects). 

The CG has been able to acquire the documentation materials through the online message boards of the 
teachers. Every student prepared a learning unit (project work) online that was subsequently discussed in the 
classroom. From the last week of April, the CG continued its activities for another two weeks as usual. 

The EG has stopped the face to face lessons in late April and enjoyed ITU online in the first week of 
May; during the second week of May these activities have been taken up in the classroom with workshop 
experiences. 

This structure has allowed us to transform the classroom into a research community in which students, 
guided by the teacher have been involved in a discovery learning, and themselves become content creators 
(Maglioni & Biscaro, 2014). In the third week of May all the students (CG and EG) were evaluated. Each 
ITU provided three video lessons created with the software Camtasia; two hypertext lessons made with the 
software Exelearning, and some concept maps created with the software Cmap tools. 

4.2 Monitoring and Assessment 

The project articulates its assessment on three levels: I LEVEL assessment of learning outcomes, II LEVEL 
assessment of the two methods (blended and flipped). III LEVEL assessment of the students perception of 
the FC model and evaluation of the approach of studying of digital students. 

4.2.1 Level 1 

During the lessons the two groups EG and CG were urged to produce deliveries, i.e. construction of some 
elaborate, plans, maps, etc. As regard the tools we used: SW Camtasia, eXeLearning, Cmaps tools, 
Mindomo, Freemind. Although both groups prepared good quality products the exchange of e-mail and 
communication with the guys in the presence of EG have been much higher. Furthermore it has been 
activated a forum (fig. 3) where students could communicate doubts and uncertainties about the path of 
learning but also to communicate with each other. 



Figure 3. TIC&DIL Forum-Chat 

Regardless from participation in the experimental model, students who participated in the pedagogical 
area workshops have taken part in a teaching workshop. The workshop setting, in fact, was the first gateway 
for direct passage from teaching (ITU) to design (Laboratory). 

Through a selection of technological resources and instructional strategies experimented in the field, it 
has allowed the students to try their hand with open applications, whiteboard (LIM) and intervention models 
for language teaching for dyslexic students. 
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This environment produced several design ideas for teaching and for technologies to support people with 
dyslexia. The student, following a bottom-up process, has been called upon to produces ideas in a design 
format that launched it to a process of shared and collaborative planning. 

Specifically, the students during the workshop activities have responded to the educational delivers by the 
teachers on the topics: after the study of the planned materials (theoretical contents and explanation of the 
use of technological tools), they have produced the papers: concept and cognitive maps (Fig. 4), audio books, 
video tutorials with the interactive multimedia whiteboard (Fig. 5 ) etc. 



Figure 4. Conceptual map made using MINDOMO (left) and using CMAP (right) 



Figure 5. Example: production of video tutorial using the IMW 


4.2.2 Level 2 

At the end of the path the final evaluation on learning outcomes for both groups started. 

The students responded to a multiple-choice questionnaire. The CG was evaluated on the program 
presented in class, the EG on the program disbursed until late April and on the ITU. It has compared: 
Learning traditional (face to face training and workshop); Blended learning (face to face training and 
workshop integrated with educational materials at distance); Flipped learning (distance learning mode 
videotaped and the next part in presence). Research shows the benefits of the FC model in university 
education about on the learning outcomes. 

The EG had more consistent performances relate to the production and evaluation. In both groups (EG 
and CG), the traditional model lesson with workshops works better if supported by blended learning. While 
the blended learning did not affect the didactic based only on face to face lectures. 

4.2.3 Level 3 

Furthermore this level evaluated the experience of teaching according to the FC and Blended Learning 
approach. The research group has drawn up and administered an Assessment Questionnaire on the flipped 
experience (QFC) (Pinnelli & Fiorucci, in press), semi structured tool organized by 40 items divided into 6 
areas (Tab. 1): 
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Table 1. Assessment Areas explored by QFC 


Assessment questionnaire on the (lipped learning experience (QFC) 
(Pinnelli & Fiorucci) 

I AREA: Socio-demographic data of the students 

Items:l-7, 21 

II AREA: Perception of the learning experience with FC 

Items 8-15,23 

in AREA: Technology assessment 

Items: 16-20,22 

IV AREA: Self-assessment of the cognitive component of the process of access 
to knowledge 

Items: 24-28 

V AREA: Self-assessment of the organizational management of access to 
knowledge 

Items 29,30,33,34 

VT AREA: Self-assessment of the social and motivational component 

Items 31,32,35,36,37,38 


The questionnaire also included two open items (39, 40) in which the students could express personal 
opinions about the strengths and weaknesses of the experience. 

This level settled out the results in terms of strategies to approach the study of a sample of 129 university 
students. 

Furthermore, this level had seen the investigation of the metacognitive skills that characterize digital 
students (see Pinnelli & Sorrentino, in press). 


5. CONCLUSION 

According to Aaron Sams and Jonathan Bergmann (2013 ) “in light of the principles of Universal Design for 
Learning (CAST, 2012) [in order to] accommodate all learners, videos, textbooks, problem sets, and other 
activities should became optional resources for learning rather than required activities. Students [should use] 
the resources that best suit them to master learning objectives”. 

As stated by Tucker (2012) teachers agree that viewing the recorded videos before class time is not 
enough to make the flipped model successful. Consequently it’s very important how the teacher creates and 
manage the didactic and the relationship with the students globally. 

The Project shows an overall very positive perception of the FC experience (see Pinnelli, Fiorucci & 
Sorrentino in press). 

The students, in fact, see in this approach a “novelty” that can stimulate, motivate them and make them 
autonomous about the learning management (time, place, tools of enjoyment). 

In contrast, however, it can induce a sense of loss, isolation, depersonalization and it can accentuate the 
relational distance among peers and with the teachers. 

In addition the research shows that this approach is strongly influenced by: a careful design process and 
planning activities (in plenary and individual study); an accurate time management; the choice of subjects; 
the ways and means of teaching contents, educational deliveries and assessments etc; Cognitive-cultural 
elements and learning and teaching styles, learning environment; technological expertise; the recovery of the 
emotional and relational aspects. 

The approach FC, beyond the involvement of innovative tools, must primarily considerate from the whole 
context of education - Teachers and students - the acceptance of a cultural learning and teaching model that 
in classroom practice, but especially in the minds, ought to be “flipped”. 
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ABSTRACT 

This paper explores the effects of a technology-enhanced pedagogical framework on collaborative creativity processes. 
The pedagogical framework is built on socio-cultural theory which conceptualizes creativity as a social activity based on 
intersubjectivity and dialogical interactions. Dialogue becomes an instrument for collaborative creativity processes 
development, and processes as: distributed leadership, mutual engagement, peer assessment or group reflection become 
triggering processes of collaborative creativity expression. Two cases studies were conducted in which two secondary 
classrooms followed a creative project. We explored the role of technology in shaping collaborative creativity processes 
and we unpacked students’ perception about the collaborative and creative processes developed during the educative 
project. Our preliminary findings showed that the technology-enhanced pedagogical framework scaffolded the 
development of collaborative creativity processes. Students reported key divergent and convergent process to solve the 
social and group challenge. And students highlighted key learning to learn processes related mostly with group reflection 
and mutual engagement. 


KEYWORDS 

Creativity, collaboration, ICT, secondary education. 


1. INTRODUCTION 

We live in changeable world, a world in motion and these changes can be seen at all levels of our life and our 
society. Many educational researchers highlight that it is under these changeable circumstances that creativity 
becomes much more present and more important than before and it is claimed to help us achieve our goals as 
individuals, as organizations, as societies (Westwood & Low, 2003). 

Creativity has been theorized from different theoretical paradigms (recent revisions: Sternberg, 2003; 
Kaufman y Sternberg, 2010). Our study is based on socio-cultural psychology of creativity, conceiving 
creativity as a fundamentally relational, intersubjective phenomenon. One of the purposes of this paper is to 
be a contribution to the social and cultural nature of creative acts and how technology can enhance these 
creative behaviours. 

In this line, our study bases on We-paradigm of creativity proposed by Glaveanu (2010). From this 
paradigm “creativity takes place within, is constituted and influenced by, and has consequences for, a social 
context” (Westwood & Low, 2003). Therefore, the role of social factors in the creative process is 
investigated. 

Our study falls on this line of research and has as a main objective the design, implementation and 
analyses of a technology-enhanced pedagogical framework for collaborative creativity in Secondary 
Education. We are interested to know, on one hand, how the pedagogical framework designed has a positive 
impact on students’ development of key collaborative processes. On the other hand, we want to study what 
role technology plays in mediating the development of these processes. 
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2. COLLABORATIVE CREATIVITY FROM SOCIO CULTURAL 
PSYCHOLOGY 

Sociocultural psychology stresses the role of the social context in shaping humans brain. Vygotsky 
(1960/1997) pointed to the importance of cultural mediation through tools and signs for the development of 
all higher mental functions. What transpires from the cultural-historical perspective is that creators use 
culturally constructed symbols and tools to produce new cultural artifacts (Moran & John-Steiner, 2003). 
Furthermore, Vygotsky was primarily interested in the ontogenesis and microgenesis of creativity and in 
creativity as a process occurring in real-life “collaborations”. 

As a synthesis, Glaveanu, (2010) highlighted three aspects that have to be considered in the socio-cultural 
psychology conceptualization of creativity, aspects that are important in designing technology-enhanced 
pedagogy: 

1 ) It considers creative acts as socio-cultural in nature and origin. Cultural traditions, social practices and 
social artifacts regulate, express, transform and permute the human mind (Shweder, 1990). There is a strong 
interdependence between individuals and their socio-cultural context. Therefore, the transactions, interactions 
and activities between these two “systems” are the origin of collaborative creativity. In Cole’s (1996) words 
the mediated action in context is the sociocultural genesis of mental functions. 

2) Socio-cultural psychology conceptualization of creativity stresses the role of intersubjectivity and 
dialogical interaction in the creative expression. Glaveanu (2010) claims that creativity is located in the space 
of interrelations. We have to investigate how creativity emerges in relations and how dialogue becomes an 
instrument for collaborative creativity processes development. The communicative and social dimension of 
collaborative creativity is highlighted by Sonnenburg (2004) theoretical framework for creating in 
collaboration. Participants have to be mutually engaged in the process of communication during the 
collaborative resolution of a task and they have to present a working style distinguished by a serious of 
dispositions that can favour the emergence of creativity in collaboration. This author highlights an open and 
free communication in which all collaborators have the same chance to contribute to the course of 
performance, and the same right that his contributions are taken seriously. Mutual trust and risk-taking are 
other key dispositions in collaborative creativity. Wegerif et al (2010) in a research in which a dialogic 
perspective of analysing creative thinking in online dialogues was adopted; claims that the creative process in 
a collaborative learning situation depends more on a tension between different perspectives rather than a 
shared framework. Thus, creative thinking emerge when further entails opposing ideas and disagreements 
being thoroughly discussed, in such a way that differing opinions and conceptions are related to each other. 
In such a process of collective learning an elaborated understanding of the learning topic can emerge. 

3) Socio-cultural psychology conceptualization of creativity looks at how cultural symbolic elements 
come to form the texture of new and creative products. Creators use culturally constructed symbols and tools 
to produce new cultural artifacts (Moran & John-Steiner, 2003). Zittoun (2007) developed the notion of 
symbolic resources; the thesis of this notion is that a group of people when facing a challenge they re- 
elaborate meaning using symbolic tools and cultural artifacts in a newly way that lead them to externalize a 
new and creative outcome. In our work, we emphasize the use of technology as cultural and symbolic tools 
that might help groups to create new artifacts to reach innovative solutions to social challenges. 

All these basic premises are at the core of the design of the technology-enhanced pedagogical framework 
for collaborative creativity presented in the next section. 


3. A TECHNOLOGY-ENHANCED PEDAGOGICAL FRAMEWORK FOR 
COLLABORATIVE CREATIVITY 

One important challenge that education has to face is how to develop a pedagogy that could promote the 
development of creative thinking of young people, and equip them to face the new challenges that the 
changeable world in which we are living is presenting every day and in the future. Our study is facing this 
challenge and has developed a pedagogical framework in which technology and collaboration have a central 
role in the developing of creative thinking. 
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Our technology-enhanced pedagogical framework is framed on socio-cultural theory presented in the 
previous section and is based on Sawyer’s creative process (Sawyer, 2010) who conceives the creativity “in” 
and “as” action. Creativity takes place over time, and most of the creativity occurs while doing the work in a 
joint activity, and is both facilitated by human social relations and improve them. 

The technology-enhanced pedagogical framework for collaborative creativity developed for this study 
involves the next seven axes and depicted in Figure 1 . 



Figure 1. The seven educative variables embedded in the technology-enhanced pedagogical framework. 

1. Challenge : The departure is proposing and defining a social challenge to solve collaboratively and 
creatively. The challenges proposed are: “Creative writing story” and “beautify a patio wall”. 

2. Solution : Students have to define a solution to the proposed challenge which will have a value for 
society, with a transformative orientation and students have to define how they are going to communicate to 
the society their transformative and valuable solution. 

3. ICT : Specific use of ICT as mediator of collaborative creativity processes. Our project used 2.0 web 
tools that allow creative organization of group ideas and enhance the construction and the structure the 
different phases of the creative process. For example, students used the tool named “Cacoo” to collect 
information, organize, structure and plan it. 

4. Collaboration : Creativity emerges on interaction with other people and with high dialogue quality. 
Social conception of creativity stands out by: a) the importance of group work: agreement of the rules of 
team work or “ground rules” (Mercer, 2000); b) the importance of dialogue and language in the creative 
communication between students. In our project we worked on these aspects as follows: 

Stimulating students into creating their own ground rules in order to promote all group members 
active participation and work. 

Giving students well-designed scaffolds to improve dialogue and better collaboration. 

5. Divergent phase - “Open the mind” : Quantitative phase for idea generation. Well-designed activities to 
promote group idea generation. For example: visualization of images in order to decide the theme of the 
“beautify the wall” project 

6. Exploration phase - “Emergency” : The students work together on the ideas that were proposed in the 
previous phase and on new relevant information. 

7. Convergence phase - “Close” the process : It refers to the moment in which the ideas and information 
are evaluated in a critic and realistic way, in order to converge in one decision and action. This phase should 
conclude with a tangible and defined product. The methodological strategies that define this phase are: a) the 
explicit argumentation of the proposed ideas; b) democratic voting and selection of the members. For 
example: students design book trailers as a way to communicate the creative pieces of writing that they 
created collaboratively. 
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4. OBJECTIVES OF THE STUDY 

1. To implement the technology-enhanced pedagogical framework for fostering collaborative creativity in 
Secondary Education. 

2. To explore the role of technology in shaping collaborative and creative processes: what technology 
students used to orchestrate their collaborative and creative project? And how students used this technology? 

3. To unpack students’ perception about the collaborative and creative processes developed during the 
educative project. 


5. RESEARCH METHODOLOGY 

Two case studies were designed in which participated two ordinary secondary classes. In the first study, 2 
language teachers and 26 students of year 8 th (13-14 years old) participated. Both teachers conducted a 
project about creative and collaborative writing. The social challenge proposed was: a real editor came to the 
school and asked for students’ collaboration on writing short stories that could be interesting to teenagers. 
Students should write the stories and explore how to rise teenagers interest on their stories.. The most 
interesting story would be published on-line in the publisher web and the group of students would have 
pocket money to spend in a popular bookshop. 

In the second study, 3 secondary teachers (one taught technology, the other taught English Language and 
the last one taught Science) participated. All teachers conducted in an interdisciplinary perspective the 
project with one group of 25 students of year 7 th (12-13 years old). The social challenge proposed was: the 
environmental school committee asks to the class to think a proposal for decorating one of the walls of the 
playground. This proposal has to be respectful with the environment and has to include the wall design, 
materials and budget, and has to be communicated to the school community in English and in Spanish. The 
proposals and their presentation would be uploaded in the school web side. 

In both case-studies students worked together in groups of 4 students. 

5.1 Data Collection and Analyses 

We collected different kind of qualitative data, however for the purposes of this paper we will base on three 
types of data: a) the work students did on computer; b) the type of technology they used and for what 
purposes and c) the responses of a questionnaire that all students answered individually. This questionnaire 
was formed for seven questions in which students were asked about their perception about what and how 
they have learnt in relation to the three main axes of our study: creativity, collaboration and technology. 

In order to inform about our second research objective - what technology students used to orchestrate 
their collaborative and creative project? And how students used this technology? - we carried out detailed 
analyses of the students’ products in the different technology they used. We will present the list of 
technology students used and some examples to illustrate the objectives students had when they used this 
technology to solve the challenge creatively and collaboratively. 

In order to inform about our third research objective -to unpack students’ perception about the 
collaborative and creative processes developed during the educative project- we carried out a detailed content 
analyses of the students’ answers in the questionnaire. A coding scheme was used to characterize students’ 
contributions. The coding scheme is presented in Figure 2. 

The coding process consisted of two steps: a) dividing students’ responses into meaningful units, and b) 
assigning a code to each unit. We decided to segment the notes into units of meaning by using semantic 
features such as ideas, argument chains, and discussion topics, or by regulating activities such as making a 
plan, asking for an explanation, or explaining unclear information. Validity and reliability aspects were 
considered in the study. 

Our coding scheme has two axes: 

a) Creativity axe. In this variable we analysed students’ awareness and perception about the two main 
creative processes: divergent and convergent (see Figure 1 ). 
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b) Collaboration axe. In this variable we analysed students’ awareness about the learning to learn together 
processes. We have characterised 4 categories: 1) distributed leadership: efficient organization of the team, 
so that all the members take the same responsibility and by exchanging roles based on that moment 
necessities; 2) mutual engagement: the members attitude based on mutual respect, opinion exchange and 
helping disposition; 3) peer assessment : altogether analysis of the work group; and 4) group reflection: 
students’ positive and participative attitude, in order to promote group wellbeing, reflection on the process 
and being able to regulate. 





Figure 2. The coding scheme 


5.2 Results and Discussion 


5.2.1 Technology Students Used to Orchestrate their Collaborative and Creative Project and 
How Students Used this Technology 

Table 1 shows the different technology students used, the cognitive categorization of its use according to the 
main objective that the students had in order to resolve the challenge in a creative and collaborative way, and 
a screenshot that exemplifies the use of the technology in our project. 

As it can be seen in Table 1, the collaborative use of the technology to solve the challenge triggered a 
certain way of processing the information and the development of specific cognitive and thinking skills. For 
example, students used Cacoo to organise and structure the information searched and collected by the 
different members of the group in order to understand it and make a significant meaning making. And 
students used Sketchboard tool design the ideas as a proposal for the elaboration of the final product. 
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Table 1. Technology used, cognitive processes promoted and example. 


ICT 

Cognitive use 

Exemplification 


Collect 

information 

locoo ■ ~~ 1 




B - - — -iH 



CACOO 


■ - -h-Mh 


Cocoo — - Qj 



■ — — — — •— — — ■ — — — 


Organize/ 

H . > . _2U- 



t 

EPOCA 

Sketchboard 

Design/ 

Elaborate 

/ ✓ ^ V 

■ - © .* ! 0 B 




Drive 

Build/ Write. 
Revise/ 
Modify. 
Discuss 

C^ptK4o 4 Hafaia ?efeeka 




Synthesize. 

Communicate. 

Design. 


Movie Maker 

NADIC SAPC 

rcmjpo WM 

QR 

Communicate/ 

externalize 

1^ JjUftnL 1^1 


5.2.2 Students’ Perception about the Collaborative and Creative Processes Developed During 
the Educative Project 

Figure 3 and 4 shows the results of each of the two cases studies analyzed and in relation to students’ 
perception about the collaborative and creative processes developed during the educative project. As it can be 
seen students are aware about the development of key creative processes related with convergence and 
divergence processes. 


Figure 3. Student’s perception creativity Figure 4. Student’s perception creativity 


Wall design project 


61% 



■ Divergence 
Convergence 


Writing project 


57% 



■ Divergence 
Convergence 
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In relation with creativity, students are more aware of the convergence processes developed during the 
project; more concretely, students stand by the importance of communication and externalization of the final 
result. The students were aware that the combination of their common ideas to create the final result and the 
design of a proper communication of the group ideas were two key processes of their creative work. Our data 
shows that the fact of communicating their projects to the school community has generated an exceptional 
leader role on students’ engagement in creative processes. Some examples of students’ answers are the next 
ones: 

“yes, by combining my team mate ’s new ideas with mine, it resulted in more original ones ” 

“make a book trailer, that is, do the story visually, and make something likeable ” 


Wall design project 



■ Distributed leadership 
b Mutal engagement 

■ Peer assessment 

■ Group reflection 



Figure 5. Interaction phases Figure 6. Interaction phases 

In terms of interaction among equals, both case studies show similar results. Group reflection is the 
subcategory that stands out the most, as it can be seen in Figure 5 and in Figure 6. Mutual engagement and 
Distributed leadership are also key processes highlighted by the students. Besides, if we look at the 
subcategories of our coding scheme, it can be seen that the subcategory “Learning Atmosphere (Group 
Reflection)” stands out, followed by Group regulation. 

We can argue that “learning atmosphere” is higher because students are aware of the importance of group 
working, and generating a good environment in order to complete their tasks creatively with a social and 
transformative value. Student have understood the importance of cresting a good working environment to 
achieve successful learnings, as opposed to the inappropriate manners that only generate conflicts (not 
listening to each other, getting angry, fight, yell, etc.) that don’t allow achieving the best learning results. 
Some examples are: 

“not getting angry ” / “not yelling ” / “not fighting ” 

“respect others ’ opinions ” 

Addressing Group regulations as a second stand out subcategory in students’ opinion, our results show 
students awareness about the importance of reaching agreements that took into account different ideas in 
order to work properly and achieve a quality learning result. Example: 

“we valued each other and we realized what we did wrong and we had to fix ” 

Through the use of collaborative dialogue, we could observe the importance that was given to the 
organization processes: how they organized, planned, regulated, etc., in order to achieve the group task in the 
best way possible. Example: 

“yes, due to the fact that we well behaved, everyone wanted to work and do it in a team. We managed to 
reach an agreement; none of us was in disagreement with another team mate ’s idea 


6. CONCLUSION 

Our study designed, implemented and analysed the impact of a technology-enhanced pedagogical framework 
in the promotion of key collaborative creativity processes in Secondary Education. 

Our study showed a positive use of technology to orchestrate students’ collaboration and creativity 
actions. In this line, the educative use of the affordances of some web 2.0 tools as Cacoo or Drive has been 
shown as powerful tools to support (a) the promotion of all students’ ideas, (b) to support the representation 
and organisation of students’ ideas in a significant way and (c) to communicate effectively among the 
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members of the group. Besides, our technology-enhanced pedagogical framework has been successful in 
developing key collaborative creativity processes. Students reported key divergent and convergent process to 
solve the social and group challenge. And students highlighted key learning to learn processes related mostly 
with Group reflection and Mutual engagement. 

Furthermore, our two case studies showed that the technology-enhanced pedagogical model supported 
students’ creation of a collective and creative solution to the challenge. Therefore, our pedagogy engaged 
students in a creative and collaborative experience in which they create new and shared knowledge, 
developed Group reflection skills and positive Mutual engagement strategies. From our perspective, our 
study shows a promising path in the technology-enhanced pedagogy for collaborative creativity. However the 
limited sample of our study does not let us to make big claims in this area, our intention is to design a more 
large scale study. 

However, students had to use different technology to get the support to develop key collaborative and 
creativity processes. From our perspective, there is a need to design technology that can support these key 
creativity processes in a same and shared workspace. Therefore our study can give powerful insights to 
technologist to design efficient technology to afford key collaborative and creative processes. 
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ABSTRACT 

The purpose of this study was to examine collaborative problem solving in a shared virtual space. The main question 
asked was: How will the performance and processes differ between collaborative problem solvers and independent 
problem solvers over time? A total of 104 university students (63 female and 41 male) participated in an experimental 
study. Participants were randomly assigned to four different experimental groups: individual & multi-tasking, 
collaborative & multi-tasking, individual & single-tasking, and collaborative & single-tasking. Results showed that the 
participants who collaborated and had multi-tasking activities outperformed the others. Additionally, collaboration helped 
to improve overall performance over time. The study offers insights for collaborative learning from both theoretical and 
methodological perspectives. 


KEYWORDS 

Collaborative learning, shared space, split attention, multi-tasking 


1. INTRODUCTION 

The collaborative learning has been defined as an instructional method that allows pairs or small groups of 
students to work together towards a common goal (Gokhale, 1995). “Collaborative learning” has been 
reviewed for the potential to enhance active exchange, critical thinking, and achievement (Johnson and 
Johnson, 1986; Totten et al, 1991). However, complex technology-mediated communications environments, 
increasingly employed for collaborative learning, is also thought to pose challenges for students who must 
split attention and engage in media-induced task switching (Rosen et al, 2013). Closer examination of the 
constructs within complex environments for shared space has been suggested as the key to understanding the 
role of the computer for effective learning collaborations in technology-mediated, virtual shared spaces 
(Roschelle and Teasley, 1995). Therefore, the present experimental study examines a complex, 
technology-based problem-solving setting for the effect on students’ problem solving while participating in 
collaborative learning in collaborative multi-tasking and independent single-tasking modes. 


2. CONCEPTUAL RATIONALE 
2.1 Collaborative Learning in Context 

Collaboration can be viewed as a process by which we negotiate and share meanings relevant to problem 
solving (Roschelle and Teasley, 1995). In the broadest terms, collaborative learning is a situation in which 
two or more people learn or attempt to learn something together ( Dillenbourg, 1999). Kaye (2012) 
characterizes collaborative learning as being a secondary outcome of a task-oriented activity. However, 
collaboration for learning does not necessarily take place simply because students are co-present (Roschelle 
and Teasley, 1995; good collaborative practice will depend on the development of reciprocity and 
cooperation among students (Chickering and Ehrmann, 1996). 
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Increasingly, communications technologies provide both synchronous and asynchronous technology tools 
that support a more abstract view of collaborative, active learning. Technology-mediated collaborative 
learning extends collaborative learning beyond face-to-face environments (Alavi and Dufner, 2005) and 
going beyond that, supporting the flexible learning paradigms that were once dependent almost solely on 
email and computer conferencing (Collis and Moonen, 2001 ). Computer-mediated communications for new 
learning paradigms are also associated with shifting philosophical foundations, from objectivist to 
constructivist views, in fields of learning theory and instructional design (Jonassen et al, 1995). Bruffee 
( 1999) points out that collaborative learning in higher education is creating a need for a reexamination of the 
role of the existing assumptions regarding knowledge, authority, and institutions within a social construction 
framework. Gokhale ( 1995) posits that collaborative learning is most effective when the primary objective of 
a teacher is not transmission of information but rather the development of a students' ability to learn. 

2.2 Collaboration for Problem Solving 

“Collaboration is a coordinated, synchronous activity that is the result of a continued attempt to construct and 
maintain a shared conception of a problem,” (Roschelle and Teasley, 1995 pp.70). Talk is the primary 
resource in the process, allowing the production of shared knowledge, divergent understandings, and 
resolution of problem-solving impediments (Roschelle and Teasley, 1995) by the use of constructive dialog. 
Collaboration in problem solving is thought to promote creative thinking skills and to reduce 
problem-solving anxiety (Gokhale, 1995). Coordination is essential to problem solving. Barron (2000) 
conducted a study to describe the types of interaction that promote coordination. He attributed differences in 
the performance of problem solving to the degree to which collaborators will have 1 ) shared task alignment, 
2) joint attention for solution, and 3) a mutuality and reciprocity of contribution. 

Within the larger framework of goals in education, the skill set required to participate in teamwork and 
solve problems collaboratively is considered a precondition for success in many learning and working 
contexts (National Council of Teachers of Mathematics, 1989; National Research Council, 1996; Rummel 
and Spada, 2005). The ability to define and solve problems is a highly valued skill in the knowledge-based, 
interdisciplinary, and distributed work of today (Barron, 2000). 

2.3 Virtual Shared Space 

The place of learning, no longer limited to a surrounding or local space, is uniquely defined for 
technology-supported collaboration. Resta and Laferriere (2007 ) characterize collaborative learning as being 
a complex concept often implemented in technology-supported virtual workspaces. Stahl et al (2006) 
examined the complexity of computer support and collaborative learning that utilize a collaborative 
communication channel bridging time and space. Wegerif (2006, p. 156) recognized changing patterns in 
contemporary work and life practices, such as student participation in online dialog in spaces of reflection. 
Teasley and Roschelle (1995) suggest that collaborative problem solving enables students to construct a 
shared conceptual structure or joint problem space. A joint problem space, which provides the benefits of 
computer-supported collaboration in a meditational framework, is dependent upon participant sharing of: 1) a 
common language, 2) a common situation, and 3) a joint activity (Teasley and Roschelle, 1995). 

2.4 The Current Study 

With the current study, we hope to obtain a better understanding of collaborative problem solving in a 
complex setting. In this regard, the first aim was to determine the degree to which the perception of 
collaboration will develop over time. We assume that collaboration will increase over the course of learning 
(Hypothesis 1). The second aim was to empirically document the degree to which collaboration would 
influence students’ performance. We assume that collaboration is positively related to the overall problem 
solving performance (Hypothesis 2). 
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3. METHODS 

3.1 Participants 

Study participants, N = 104, were university students (63 female and 41 male), who were enrolled in an 
intermediate-level teacher education course. Their mean age was 23.49 years ( SD = 4.22). 

3.2 Materials and Instruments 

The current experiment had students solve analytical reasoning (AR) problem tasks that are samples of 
Graduate Record Exam (GRE) problems. As shown in Table 1 below, AR problems require an understanding 
of a given structure of arbitrary situational relationships for subsequent deduction of new information from 
the given relationships for constraint-satisfaction (Kaufman et al., 2001). The AR problem questions were 
assigned a difficulty rating, on a scale of 1 to 6, where 1 = least difficult and 6 = most difficult, based on 
guidelines for analysis of content characteristics to the difficulty and discrimination of GRE problems 
(Chalifour and Powers, 1989). 


Table 1. Two examples of sample GRE reasoning tasks from lowest and highest difficulty levels, and solutions 


Task 

T_dl 


T_d6 


Example 

Four of the following five are similar in a definite way and so form a group. 
Which one of them does not belong to the group? 

A. Umbrella 

B. Gloves 

C. Shirt 

D. Shoes 

E. Cap 

A pesticide producing company states that their unused pesticide that gets 
dumped does not pose a threat to the aquatic life in the surrounding area. If 
this is correct, then why have local fish been dying in this region? Due to 
the fact that the pesticide company is not located in a highly fish-populated 
area, they implicitly admit that the pesticides they produce are relatively 
dangerous to the nearby aquatic life. 

Of the following statements listed below, which one would be most likely 
to weaken the argument of the author if it were true? 

A. The possibility of pesticides filtering into the local water region was 
underestimated in the past. 

B. Funds for environmental company cleanup, which concern waste dumps 
that are poorly run, are reserved for rural regions only. 

C. It would be pointless to locate chemical dumps where they would be 
most harmful, unless they can be proven 100-percent safe. 

D. Dumps that are located in areas without large fish populations have 
fewer government interventions and are also less expensive. 

E. City people are most probable to sue the company if the dumps cause 
them health problems. 


Solution 


A. Umbrella 


D. Dumps that are 
located in areas 
without large fish 
populations have fewer 
government 
interventions and are 
also less expensive. 


Multiple instruments were administered to test participants’ pre-dispositions and changes of dispositions 
over time. These include the following: 1) Verbal ability test (r = .96; split-half reliability; Amthauer et al, 
2001); 2) Multi-tasking preference inventory or the Inventory of Polychronic Values (IPV; Bluedom et al, 
1999). The IPV has 10 items measured on a 7-point Likert response scale (anchored from strongly disagree 
to strongly agree) with higher values indicating a more polychronic or multi-tasking attitude. Conte et al 
(1999) reported Cronbach’s alpha=.822 and higher as evidence for construct validity for this instrument. 
The retest-reliability coefficient over a 2-month interval is .78 (Conte and Jacobs, 2003); 3) Integrated 
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communication technology learning (15 items, Cronbach’s alpha = .605); 4) Formal-to-informal learning 
scale (12 items, Cronbach’s alpha = .695; Author, 2014); 5) Technology affinity scale (22 items, Cronbach’s 
alpha = .624; Author, 2013; and 6) Confidence, effort, motivation, collaboration, tools, and strategy 
inventoiy (5 items, Cronbach’s alpha = .692). 

3.3 Analysis 

Initial verbal ability and multi-tasking preference scores were calculated for each student. Each participant’s 
task solution scores were determined at each measurement point. Additional measures were also calculated, 
including confidence in accuracy of the solution, level of motivation, the problem-solving strategy applied, 
an estimated of time on task, the estimated degree of collaboration, and the method of collaboration. 


4. RESULS 

4.1 Development of Collaboration 

We computed a repeated-measure MANOVA with the intensity of collaboration at five measurement points 
as a within-subjects factor, and experimental groups (CMT, CST) as a between-subjects factor. MANOVA 
revealed a significant main effect of time on intensity of collaboration, Wilks’ Lamda = .782, F( 4, 47) = 3.28, 
p < .05, if = .218, and for time x group, Wilks’ Lamda = .771, F( 4, 47) = 3.49, p < .05, if = .229. The 
sphericity assumption was met (jf ( 9 ) = 11.45, p = .25). The difference between measurements was 
significant, F( 4, 200) = 2.43, p < .05, if = .046. 

A pairwise comparison of intensity of collaboration at each measurement point (MP) indicated significant 
differences between experimental groups as follows: MP2, ?(50) = 3.61, p < .001, d = 1.00; MP4, ?(50) = 
3.17, p < .01, d = .88 ; MP5, f(50) = 4.64, p < .001, d = 1.29 (see Table 2 for descriptive statistics). 


Table 2. Means (standard deviations in parentheses) of intensity of collaboration over time 


Exp. group 



Measurement point 



MP1 

MP2 

MP3 

MP4 

MP5 

CMT (n = 26) 
CST (n = 26) 

4.08 (2.17) 
3.92(1.77) 

6.19(2.06) 
4.15 (2.01) 

5.15 (2.03) 
4.19(1.96) 

5.77 (2.63) 
3.62 (2.26) 

5.54(1.96) 

3.12(1.80) 


Note. CMT: collaborative & multi-tasking; CST: collaborative & single-tasking 


Further, we found a significant interaction effect of time and group on the intensity of collaboration, 
F( 4, 200) = 2.88, p < .05, if = .054. Figure 1 shows the interaction effect on the intensity of collaboration. To 
sum up, results showed that the participants who were confronted with multi-tasking activities outperformed 
the others. Accordingly, we accept Hypothesis 1 . 
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Experimental 

group 

CMT 

CST 


Figure 1. Interaction of experimental group x time on collaboration 

4.2 Influence of Collaboration on Performance 

The regression analyses results for the acceptance and use of the three examples of learning analytics systems 
(ALA) on problem solving performance are presented in Table 3 yielding a AR ' 2 of .319. Clearly, 
collaboration positively predicted the problem solving performance, indicating that the higher the perceived 
collaboration, the higher the overall problem solving performance. Accordingly, we accept Hypothesis 2. 


Table 3. Regression analysis predicting collaboration on problem solving performance 

R 2 AR 2 B SEB p 

.333 .319 

Collaboration .520 J04 ,577*** 

Note. *** p < .001 


5. DISCUSSION AND CONCLUSION 

This study examined how technology-supported collaboration would develop over time, as well as the effects 
of collaboration on problem-solving performance. The results of the study showed that 1) the participants 
who were confronted with multi-tasking activities outperformed those who did single tasks; 2) the higher the 
participants perceived collaboration, the higher they demonstrated overall problem solving performance. 
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In general, students became increasingly more collaborative over time, and collaboration was a strong 
predictor for overall performance. 

This study is significant in several ways. From a theoretical perspective, with the increasing complexities 
of learning environments being afforded by new technologies, it is important to examine aspects of 
collaborative learning and problem solving in flexible and multitasking environments. Methodologically, it is 
important to advance an understanding in this area of learning by testing hypotheses and conducting 
experiments to obtain results that may assist educators and learning technologists to advance understanding 
of how best to design and support a student’s ability to coordinate, collaborate, and problem solve in new 
distributed workspaces. 

Helping students to “develop their capacities for productive engagement in collaborative problem solving 
is both an educationally and socially important venture” (Barron, 2000 p.433). The new spaces of study and 
work are increasingly virtual and visited by individuals who are distributed in time and place (Resta and 
Laferriere, 2007). These technology-supported workspaces enable new models of flexible collaboration for 
learning and problem solving, although they could potentially increase cognitive overload as well. It is 
important to examine elements and dynamics of such workspaces to ensure smart learning environments for 
the learners. 
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ABSTRACT 

At present social networks are becoming important in all areas of human activities. They are simply part and parcel of 
everyday life. They are mostly used for advertising, but they have already found their way into education. The future 
potential of social networks is high as it can be seen from their statistics on a daily, monthly or yearly increase in the 
number of their users. The purpose of this article is to provide a short description of the concept of social network and its 
classification according to the way how they function or their main purpose. In addition, the authors discuss two main 
approaches to teaching and the educational theories out of which connectivism is the first theory which takes into account 
the existence of computer networks and considers all knowledge and abilities as a result of mutual interconnection of 
information and people. Thus, it is the theory which enabled the implementation of ICT into education and has 
completely changed the traditional teaching and learning. In conclusion, the authors also explore a few already existing 
educational social network sites and their benefits for education. 


KEYWORDS 
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1. INTRODUCTION 

At present social networks are becoming important in all areas of human activities. Every man is a member 
of social network when he is born in the relevant community, i.e. in the family. Later, man comes in contact 
with the so-called Internet Social Networks (SNSs). The object of SNSs can be just a classic e-mail or more 
complex web tools aimed at service promotion, sharing, information search and communication among their 
users. 

Social network can be perceived from different points of view. From the sociological point of view, social 
network is a connected group of people who can influence one another. In most cases the group is based on 
common interests. It can also originate for family reasons when the group members are relatives. The concept 
of social network was used already before the rise of the Internet. It was defined by the sociologist J. A. 
Barnes in 1954. Barnes was researching the relationships among the Norwegian fishermen. Thus, the primary 
meaning of the social network was sociological which served to the description of social structures connected 
for different reasons (Pavlicek, 2010). Barnes defined the social network as a set of points and some of these 
points are mutually connected by lines (relationships - bonds). They afterwards form a total network of 
relationships, the so-called social network (Rulf, 2013). 

According to Pavlicek (2010), the social network contributes to mutual influencing and enrichment of the 
whole group. He claims that the group identity accompanies a person all his life and he usually becomes a 
member of more than one group at the same time. And in each group he behaves and communicates 
differently. 

With the arrival of new information and communication technologies, the social network acquired a 
completely new meaning. The advantage of the Internet social network is that people can communicate 
independently on their location and time, and thus keep in touch anywhere. Moreover, they have easier 
access to contact their friends and find information about them. 

The online social networks have many definitions. One of the most frequently used is provided by Boyd 
and Ellison (2007) who define SNSs as web-based services that allow individuals to construct a public or 
semi-public profile within a bounded system; articulate a list of other users with whom they share a 
connection; and view and traverse their list of connections and those by other within the system. The Czech 
expert in social networks Bednar (2005) defines the online social network as follows: The online social 
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network is a system which enables to create and maintain a list of mutually connected contacts, friends. Each 
user of such a system has his own characteristic features which are publically available to other users. Within 
this system people can search for one another and form a virtual community in this way. A more advanced 
form is the browsing of the list of friends. The social network also enables searching for former colleagues or 
virtualization of working teams. In addition, there is a possibility to upload different kinds of information or 
photos. 

Since 2003 there has been a myriad of social networks out of which some are really popular nowadays 
such as MySpace, YouTube and, of course, Facebook (Pavlicek, 2010). Consult Fig. 1 for the historical 
milestones in the development of social networks. 



Figure 1. Historical milestones in the development of social networks 


2. CLASSIFICATION OF SOCIAL NETWORKS 

Social networks can be divided into several types according to the way how they function or what their main 
purposes are (Platko, 2010). Among the principal categories there are social networks based on the profile, 
content, the so-called white label networks, microblog or virtual networks (cf. Childnet International, 2008). 
Fig. 2 provides a graphic illustration of the classification of SNs. 
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Figure 2. Classification of social networks 


Profile based social networks - these are the networks in which it is important to interact with people. The 
personal profile has its significant role in this because thanks to it, people can keep in touch with other 
people, share their opinions, photos or some important events (cf. Young People and Social Networking 
Services, 2005-2015). The most popular networks of this type are undoubtedly Facebook, followed by 
Google+ (cf. Cerna and Cerny, 2012). Also the professionally oriented social network Linkedln is among 
these SNSs. 
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Content based social networks - the personal profile is not that much important in this case. It is the 
content which is the key and is shared by people on the social network. This social network can include 
videos, music or pictures. These media are usually accessible to all, including those who are not registered on 
these SNSs. Probably, the best known representative of these SNSs is YouTube which enables users to record 
and share their videos. The portal which aims at photos and is worth mentioning is the mobile application 
Instagram. 

White-label social networks - this is a kind of services which offer an independent development of social 
network on the basis of the offered platform. Thus, users can create their own mini community, i.e. their 
small social network. They can make such a social network which would meet their expectations and needs. 
Thus, they can generate their own Facebook or Twitter in this way. Another name for this social network is 
Private label (cf. Owyang, 2007 ). The representatives of these platforms are also PeopleAggregator or Ning. 

Microblog social networks - the main feature of these SNSs is publishing of short messages with a 
chance of adding a video or a picture. This is displayed to all subscribers of a given user. This type of social 
network is predominantly used on mobile phones and particularly, in case of worse text messaging (cf. Cerna 
and Cerny, 2012). The main representative of these SNs is Twitter. This network offers a platform both for 
mobile phones and normal web browsers. The network is becoming quite popular in the Czech Republic. 
Another microblogging tool is Tumblr.com. 

Multiuser virtual networks - these are the categories which are borderline social networks. Their purpose 
is to enable users to communicate among one another, but not with the help of their profile but with the help 
of Avatars. Most of these networks are game servers such as World of Warcraft. 


3. POPULAR SOCIAL NETWORKS AS TOOLS FOR EDUCATIONAL 
SUPPORT 

The most popular social network nowadays is undoubtedly Facebook. The user who wants to use Facebook 
services must be 13 years old and above. Therefore the application of this SNS in education is possible for 
the students of higher classes (8th and 9th grades of the elementary school). Schools often use Facebook as a 
tool of their presentation in public. Students form groups on Facebook whose size corresponds to the number 
of students in one class. On this SNS they share study materials, current information about the assigned tasks, 
tests or other assignments and events. 

Twitter also offers practical exploitation during tuition at school as well as at home. Twitter as an 
educational tool can serve for communication, organization, searching for sources or written communication 
(Staff Writers, 2011). Some teachers at Northern Illinois University in the USA use Twitter as a noticeboard 
on which they record current information concerning their classes, for example, cancelling their classes. 
Therefore, this SNS can be used for setting teacher’s consultation hours. Twitter can be also used as a note 
blog for each class where students can upload their contributions. Students can debate there online and thus, 
they can receive almost immediate feedback on their discussions. The teacher can ask his students to inform 
him about what they are learning and what learning difficulties they are having by the so-called tweeting. In 
this way students create their online diary of learning. In addition, parents can exploit Twitter for an 
overview of their children’s activities. Twitter can be also used for summarizing of the teaching matter on a 
given day. Moreover, Twitter provides their users with an extensive source of information. Students can with 
its help communicate with experts. Twitter can also broaden student’s horizons, their vocabulary, spoken and 
written communication. As Twitter has a limited length of messages, it is also a suitable tool for concise and 
simple expression of user ideas. 

YouTube EDU is a special educational channel focused on the educational videos. It is the so-called 
global video classroom which users can exploit both for school or home instruction. The channel offers 
interesting videos from all fields of human activity. Users can sort them out not only according to their 
interests but also according to their level of education. The videos are mostly made by the representatives of 
various educational institutions or experts in a given issue (YouTube EDU, 2013). Furthermore, teachers can 
modify the content of the service to their teaching purposes. The content is divided according to individual 
subjects and difficulty level (YouTube pro skoly, 2013). 
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4. CHANGES IN THE EDUCATIONAL PROCESS 

With the development of ICT, the educational issues have become even more topical. When studying the role 
of technologies in teaching, two approaches are common: instructive and constructive. In case the teaching is 
controlled directly by computer, it is the instructive approach. If the activity is based on student’s own 
initiative, it is the constructive method (Brdicka, 2008). 

Instructive approach 

• Fixed syllabuses and standards 

• Learning by drilling 

• Everybody does the same 

• Testing and grading 

• Teacher is the highest authority 

• Teacher is a source of information 

Constructive approach 

• Thematic learning plan 

• Learning by understanding the connections 

• Individual or team assignments 

• Verbal evaluation 

• Teacher as a facilitator and coach 

• Anybody can be a source of information 

In practice the instructive approach prevails. The student is checked by the teacher. However, with the 
fast development of technologies, the constructive approach is becoming foreground. In this approach the 
student is responsible for his activities and learning (Brdicka, 2004). Nevertheless, education cannot be 
defined only by two approaches. There exist far more conceptions, out of which the following three are the 
most frequent: behaviorism, cognitivism and constructivism, which had been developed before ICT started to 
have an impact on education. But in spite of these theories, a new theory for suitable implementation of ICT 
into learning environment was needed. And it was G. Siemens who came up with such a theory, called 
connectivism (Siemens, 2004). He points out at the limitations of those three theories, emphasizing that the 
central idea of these conceptions is that learning occurs only inside a person and the fact that the learning can 
occur outside the person as well is omitted. Moreover, he is critical of the fact that learning theories are just 
focused on real process of learning, not on the value what is being learned. Connectivism or the theory of 
education in the social network environment (Brdicka, 2008) has a direct impact on education (Downes, 
2014). Thus, connectivism is the first theory which takes into account the existence of computer networks 
and considers all knowledge and abilities as a result of mutual interconnection of information and people 
(Brdicka, 2011). 

Siemens (2004) has formulated connectivism on the basis of the following eight basic principles: 

• Learning and knowledge rests in diversity of opinions. 

• Learning is a process of connecting specialized nodes or information sources. 

• Learning may reside in non-human appliances. 

• Capacity to know more is more critical than what is currently known. 

• Nurturing and maintaining connections is needed to facilitate continual learning. 

• Ability to see connections between fields, ideas, and concepts is a core skills. 

• Currency (accurate, up-to-date knowledge ) is the intent of all connectist learning activities. 

• Decision-making is itself a learning process. Choosing what to learn and the meaning of incoming 
information is seen through the lens of a shifting reality. While there is a right answer now, it may 
be wrong tomorrow due to alternations in the information climate affecting the decision. 
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Part of the learning process according to the theory of connectivism is the so-called Personal Learning 
Network (PLN). Each pupil creates his own PLN through which he is assessed and evaluated. In fact it is the 
network which is made by people with whom one shares his experience and knowledge, receives new 
information or provides opinions. 


5. DISCUSSION 

As Bloom (1956) states, learning is a process helping pupils in their development from the lower forms of 
thinking (e.g., knowledge or understanding) to the higher forms of thinking (e.g., synthesis or evaluation). 
And the social networks can assist in reaching these forms. For instance, Flickr can help in the acquisition of 
knowledge, Wikipedia in understanding this knowledge and YouTube is suitable for the synthesis. 

In addition, social networks can bring many other benefits for all stakeholders such as teachers, students 
and parents. The Internet social networks are good tools for easier understanding of economic, political and 
social events. Another advantage is their easy access, which is independent on location and time. Students 
can log in a social network from cosines of their home and consult all the necessary school issues with their 
peers and teachers, particularly when they fall ill. The social network may also improve mutual relationships 
among students. 

At present there is a sufficient offer of the social networks which were created to cover both teachers’ and 
students’ needs not only during classes but also in case of home learning. These include, for example, itrida 
(iclass = the Internet class), Edmodo or Schoology. The iclass focuses on basic and secondary schools. At 
these portals teachers and pupils can find a lot of learning materials. The use of the platforms is free of 
charge (Krizko, 2013). Particularly Edmodo, which has been in use since 2008, was developed for teaching 
purposes. Its environment reminds of Facebook. Therefore its users do not have any difficulties in controlling 
this educational environment (Strnadova, 2012). While working in these platforms, users can be divided into 
virtual classrooms where they share their notes or materials (consult Fig. 3 for an example). The most 
extensive is, however, Schoology which offers the most functions (Moldrik, 2013). The only disadvantage 
for a Czech user is that the last two SNSs are run only in English. 
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However, Rambouskova (2015) in her study explored which from the three social networks described 
above Czech pupils at elementary schools would like to use in their classes. On the basis of their responses, 
she discovered that they favored Edmodo because it was a good source of information and working in this 
social network was very attractive. Besides, those pupils who were ill could access their classes online and 
thus, they did not miss any lessons. 


6. CONCLUSION 

Thus, technologies in the 21st century and a new theory - connectivism, which enabled their implementation 
into education, have completely changed the way of teaching and learning. Nowadays, SNSs are part and 
parcel of everyday life. They are mostly used for advertising, but they have already found their way into 
education. The Internet social networks proved to be beneficial for education mainly for the following three 
reasons: easier communication between (teacher -pupil; pupil-pupil; teacher-teacher); improvement of class 
relationships; and immediate access to information. 

The future potential of SNSs is high. It is expected that SNSs will become a common tool in education 
which will be used at each educational institution. In addition, there are already a few research studies (e.g., 
Bosch, Preez and Michell, 2009 or Wang et al., 2012) in the field of social networks which prove that SNSs 
can be very effective learning tools that improve students’ performance and motivation to learn. 
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ABSTRACT 

The vast majority of research in educational technology focuses, justifiably, on what might be described as "short-term" 
(or perhaps "medium-term") questions: how to improve an existing software system, how to assess a particular classroom 
innovation, how to teach some current subject matter in a more effective fashion. From time to time, however, it is worth 
stepping back from such questions and taking a longer view of children's technology: what are the larger patterns by 
which certain technologies become associated with children's work? In this paper, we examine a broad thematic pattern 
through which "adult" (or "professional") technologies become progressively associated with children's activities. As an 
example of how this analysis can be put to use for future design, we describe early steps in an effort to adapt a 
particularly powerful manufacturing technology ("pick-and-place") for children's crafts. 


KEYWORDS 

Educational computing, educational 3D printing, children's pick-and-place device. 


1. INTRODUCTION: TECHNOLOGIES FOR CHILDREN S WORK 

In the main, when researchers write about educational (or children's) technology, they have a specific 
technology in mind (e.g., desktop computers, handheld devices, Arduino microprocessors, to name just a 
few). In the same vein, these researchers tend to have a specific goal in mind as well: how to improve the 
teaching of geometry, how to make a particular interface more usable, how to teach children to program. This 
approach is a staple of educational technology research, and deservedly so: it represents the means by which 
steady progress is made in the field at large. 

At the same time, however, it is occasionally worthwhile to stand back and take a longer view of 
children's technology-to look at patterns that play out over a period of decades or longer in the landscape of 
tools, techniques, and materials available to children. From the vantage point of the present, children are 
often described as "naturally drawn" to devices such as computers: there is a default assumption, in adult 
discourse, that children acquire a level of comfort with novel technologies that is unavailable to their 
hidebound elders. It wasn't always so, at least in the case of computers. In John Markoffs [2005] book What 
the Dormouse Said, focusing on the early history (and pre -history) of the personal computing era, he writes 
about a former aerospace engineer. Bob Albrecht, who taught Fortran programming to children in the 1960's: 

Along with his other chores, [Albrecht] began to teach a small group of high school students how 
to program... The class became extremely popular, and soon the University of Colorado was 
offering an extension program that involved more than one hundred high school kids. Albrecht 
took his class on tour, at one point accompanying some of his students from the original Denver 
school to a National Computer Conference meeting. There they demonstrated their programming 
skills on the CDC 160 machine, shocking the high priests of computing. At the general conference 
meeting, there were subsequent complaints that someone had even considered turning children 
loose on computers! [Markoff, p. 181] 

This anecdote is worth quoting at some length because it is remarkable to reflect on how the conventional 
wisdom has been almost completely inverted in a mere half century. The attitude of Markoffs "high priests" 
was born of an assumption about what technologies were appropriate for adults-more specifically, for adult 
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professionals-and what technologies were "for children". Since computers were highly expensive 
instruments, programmed at the time by trained technicians, it was unthinkable for children to use them. 

Before we take too condescending an attitude toward the prejudices of the "high priests", we might wish 
to reflect on our own present-day assumptions of which technologies belong, or do not belong, to the realm 
of children's work. For the computer professionals described by Markoff, setting children loose on computers 
was anxiety provoking as, among other things, a matter of cost: these things are expensive! What if the kids 
touch the wrong toggle switch and (heaven forbid) break them? Looking at the current landscape of 
professional technologies, it is not hard to ferret out similar reflexes within ourselves. Should children, for 
example, be permitted to play with a high-speed camera? With an electron microscope? With a 
supercollider? Note that in such scenarios, the immediate concern is not primarily for the safety of the child : 
we don't expect that the child will be hint by (say) the electron microscope toppling on her. Rather, the 
concern is for the safety of the device. A child working with an electron microscope? Absolutely not: these 
things are expensive. 

The purpose of this paper is to explore, at least in an initial and tentative way, several of the issues raised 
by Markoffs anecdote, and by reflections on the long-term development of children's technology. To state 
our thesis briefly: there has been a recurring pattern over the past century or more of technological transition, 
from professional or industrial technologies to children's technologies. The most prominent example of this 
pattern is the computer, but the same thematic trends can be identified with other technologies-the camera, 
the color inkjet printer, and (a recent and striking case) the 3D printer. Repeatedly, technologies that were 
formerly associated with specialized professional training have come to be used by children. In many cases, 
the barriers to be crossed were both economic (making the technologies cheaper) and cultural (re -imagining 
the potential interests and abilities of children). It is our belief that by examining these historical patterns, we 
can spark our imagination toward the creation of novel, unexpected technologies for children's expressivity. 
In the final section of this paper, we outline our own early steps toward employing this design heuristic for 
the creation of a novel, child-focused version of the (hitherto industrial) pick-and-place machine. 


2. THE TRANSITION FROM PROFESSIONAL TO CHILDREN S 
TECHNOLOGY: KEY THEMES 

The previous paragraph sketched the foundations of a recurring pattern in the evolution of children's 
technology. In this section, we expand on that basic pattern by finding commonalities and contrasts with the 
work of other researchers. 

2.1 Leisure-oriented Technologies for Adults 

In one sense, the appropriation of professional technologies by youngsters might be interpreted as simply a 
by-product of a larger phenomenon by which such technologies are "democratized" more generally (cf. 
Mumford [1934], p. 278). After all, not only do modern-day children use computers, but so do adults of all 
ages, and for an endless variety of (serious and "non-serious") purposes. Thus, we might argue that it is 
primarily adults, not children, that have led the way in the de-professionalization of computing technology. 
In the more recent case of 3D printing, children are being introduced to the technology in tandem with its 
adoption by the "maker movement" of hobbyists and amateur builders. In such cases, then, we might argue 
that the real phenomenon of interest is the (adult) movement of professional activities toward independent, 
leisure-oriented uses. 

Some of these themes are suggested by Rachel Manes' [2009] Hedonizing Technologies, an indispensable 
book-length treatment of the transition of certain activities from industrial to leisure contexts. Manes' focus is 
almost exclusively on adults, with a focus on the (primarily female) audience for textile crafts such as 
knitting and embroidery. Manes coins the term "hedonizing" to denote the process by which formerly work- 
related activities become pursued primarily for fun, as hobbies. Her discussion of the process is especially 
perceptive, and suggests some avenues of continuity with children's technology: "Unlike industrial workers, 
hobby artisans have complete control over what tools and materials they use, and since efficiency of 
production and marketability of product are rarely issues, they buy or make the tools and materials they most 
enjoy using. Leisure theory and sociological sources illuminate the kinds of pleasures artisans experience in 
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their craft, which include control and mastery, the beauty or the individuality of the product, the sensual 
enjoyment of the task.. .the escapist quality of immersion in voluntarily chosen work, and socialization 
opportunities with other crafts enthusiasts." [Manes, 2009, pp. 13-14] 

Clearly there are parallels here with the transitions common to children's technology; but there are also 
interesting points of difference. For one thing, as Manes notes, many of the hedonized technologies whose 
history she traces are in fact not state-of-the-art industrial technologies, but rather old-fashioned technologies 
made obsolete by the process of high-volume industrialization. Thus, leisure-oriented textile work 
represented a pleasurable return to an older, pre-factory style of activity; and one can identify (as Manes 
notes [p. 122]) an element of nostalgia in at least some instances of democratized technology. 

With children, the emphasis is different. Youngsters, after all, are not particularly given to nostalgia; and 
often, the cultural emphasis in adapting professional technologies for kids is expressly to celebrate the very 
novelty of the technology, and quite possibly the presumed job skills associated with its mastery. (No parent 
would object to his child’s playing with 3D printers on the grounds of the devices' uselessness or 
obsolescence.) Moreover, even when focusing on (say) computers, the re-thinking of the professional 
technology has a somewhat different flavor when the audience is composed of children. A particularly vexing 
element is the sometimes tense relationship between true leisure activity for children-what one might call the 
"hedonized" version of children's computing— and school or classroom usage. A "child's computer" might 
look like a somewhat different instrument when associated with home or hobbyist usage and with classroom 
usage. Adult leisure technologies rarely exhibit this kind of dual vision in the lives of their users. 

2.2 Consumer Technologies and Creative Technologies 

A second theme worth touching upon in this discussion is the distinction between certain types of consumer 
technologies for children (notably television, electronic toys, and video games), and technologies that, 
adopted from professional settings, thereby expand the presumed capacities and expressive range of children. 
To interpret the distinction in a somewhat broad-brush way, the advent of children's radio (cf. Cross [1997], 
p. 107) or television shows, while of historical interest, is not an especially apt example of the pattern we are 
focusing on in this paper. In these consumer-oriented cases, some advanced technology is adopted primarily 
for children's play, entertainment, or education; but there is no reason to think that children are modeling or 
re-inventing professional or adult creative activities for themselves. 

To put the distinction in a slightly different way, and focusing on the example of television: adult anxiety 
about children's television might be said to focus on (mental or emotional) safety concerns. Is the 
programming appropriate for children? Are shows too violent or disturbing? Are they harming the child's 
attention span? These are the sorts of arguments that have historically raged around children's television (and 
have been revived in the case of video games), but they have a different flavor than the anxiety of Markoffs 
earlier-mentioned "high priests", who were focused on the safety of the device. No one worries that a child 
watching television is potentially risky for the TV; and by and large, the worry is that television will "dumb 
children down", rather than expect too much of them. 

For an illuminating contrast with children's consumer technologies, consider the physicist Freeman 
Dyson's [2015] advocacy for placing the tools of current-day biotechnology in the hands of children: "The 
final step in the domestication of biotechnology will be biotech games, designed like computer games for 
children down to kindergarten age but played with real eggs and seed rather than images on a screen. Playing 
such games, kids will acquire an intimate feeling for the organisms that they are growing." [Dyson, 2015, p. 
3] Dyson's vision may or may not sound desirable (or practical) to some readers, but it fits squarely within 
the (anxiety-provoking) pattern of adopting advanced professional technologies to empower children. 

2.3 Fantasy (Imitative) Technologies and Child-adapted Technologies 

A final theme worth mentioning here is that there is a subtle, and not always clear-cut, distinction in this 
discussion between children's technology as unexpectedly appropriated by children (as in the case of the 
computer, or 3D printer), and children's artifacts that imitate adult technology (as in, say, toy cars, or faux 
kitchen appliances, or model erector sets). Historically, many toys of the previous century have been based 
on the assumption (often realistic) that children wish to mimic the work and activities of their elders; thus, a 
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small child might pick up a toy plastic saw and pretend to do "real" carpentry. This sort of design might be 
called imitative design in that it allows or encourages children to pretend to do adult work. 

Imitative design is a factor in many children's toys and games; in such cases, technology ( where it occurs) 
is generally imaginary, fantasy-oriented, or miniaturized. A fantasy toy oven, for instance, might be non- 
operational, or operate only at safely low (light-bulb-produced) temperatures. In contrast, children's 
computers are real computers; indeed, they are machines of far greater power and versatility than those that 
Markoffs "high priests" themselves used. Similarly, the adoption of such devices as 3D printers for children 
represents a true democratization of technology: the printers might be less powerful and high -resolution than 
their professional cousins, but they are most definitely real printers. 

There is much more to say about these themes (and others left unmentioned), but for now we can sum up 
the implications of our discussion for design: namely, that we can look to the professional or industrial world 
for powerful technologies that can be adapted, without excessive loss of power or performance, to the 
creative activities of children. The express goal of this sort of design is to expand the creative and intellectual 
capacities of children— to make their worlds more enjoyable and challenging-while taking due account of the 
cognitive and physical limitations (and sometimes advantages) of children vis-a-vis adults. The final section 
of this paper presents an example of the sort of novel design project that we have in mind. 


3. AN OPPORTUNITY: A CHILDREN S PICK-AND-PLACE DEVICE 

The previous discussion might conceivably be interpreted as of "theoretical" interest only: having identified a 
pattern in children's technology, what can we do with it? Our argument is that surfacing such a pattern can 
act as a springboard for design. As a consequence, we have recently begun work on a child -friendly version 
of a hitherto "professional" computer-controlled device: a pick-and-place machine. In industrial settings, such 
devices are employed to rapidly place objects (such as electronic components) onto surfaces with high 
precision; in essence they are automatic assembly devices for complex multi -part objects. In our " 
child-oriented" version of pick-and-place, we envision a device that children can program, and through which 
they can rapidly create complex 3D constructions made of standardized sets of small pieces (such as 
hexagonal chips, cubic elements, or mosaic tiles). The device is still in its earliest stages (and a paper devoted 
to its operation is in preparation); but crucially, the impetus for its creation is precisely due to the historical 
reflections described in this paper. Our goal, in keeping with the earlier discussion, is to place in children's 
hands a device that can tastefully expand and challenge their creative range. Pick-and-place is just one 
example of this design heuristic; one might likewise look to the design of novel materials, or (with Dyson) 
biotechnology as a source of professional technology for adoption. In any event, the transition from 
professional to children's technology should provide a rich source of inspiration for future designers. 
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ABSTRACT 

Formal sex education is a key strategy to help prevent unplanned teenage pregnancies, sexually transmitted diseases, 
sexual abuse and social discrimination. However, research highlights human sexuality is a difficult issue for educators to 
communicate with young people in traditional class settings. The growing tendency for young adolescents to 
communicate in online social settings presents opportunities to explore whether technology may be employed in this area. 
This paper presents a study investigating the use of a virtual learning environment to support communication between 
teachers and students in a Relationship and Sexuality Education class setting. 


1. INTRODUCTION 

Relationship and Sexuality Education (RSE) in schools has become an internationally important issue given 
the prevalent global rise of HIV/AIDS, sexually transmitted diseases (STIs) and unplanned pregnancies 
among the 15-24 age group (WHO, 2006). Barriers to providing comprehensive sexuality education such as 
high level of discomfort among teachers and students in discussing sensitive sexual issues (Mayock et al; 
2007) and lack of communication associated with inhibition experienced by both cohorts (Hyde & Howlett, 
2004) persist (Selwyn & Powell, 2007). Lester & Allan (2006) argue that the discomfort some teachers 
experience delivering sex education results in textbook teaching and reluctance to answer questions or enter 
into discussions with students. This is compounded by the fact that the language and terminology used by 
teachers in the classroom is often not the same as the language of adolescents, and this in itself can create a 
communicative barrier in the teaching of RSE (Mannix McNamara & Geary, 2003). Given that 24% of girls 
and 30% of boys have had sexual intercourse by 15 years of age (WHO: 2008), it is essential that we ensure 
all adolescents get informed sex education. 

Kanuga & Rosenfeld estimated that 75% of youth had used the Internet to look up health information and 
44% to search sexual health information (2004). Although computer based sexual health websites are widely 
available, the ease of finding sexual health information on the web has raised several issues of concern such 
as: inaccurate or irrelevant information; lack of interactivity; and potential harm due to the ease of 
adolescents to stumble across sites with nefarious intention (Kanuga & Rosenfeld. 2004). This work presents 
a study on the development and adoption of a school-based virtual learning environment (VLE), moderated 
by qualified counselors, to tackle the difficulties of communication between students and teachers of RSE, 
and evaluate its effectiveness in overcoming barriers of communication presented by traditional classroom 
teaching. 

1.1 Computer Mediated Communication (CMC) 

CMC is communication mediated by interconnected computers, between individuals or groups separated in 
space or time (Luppicini, 2007). Its primary advantage is its potential to enhance communication by 
bypassing limitations of temporal and spatial constraints. The mechanism through which CMC brings about 
enhanced communication includes asynchronous and synchronous communication capacity, high 
interactivity and multi-way communication (Lee, 2010). While face-to-face (f2f) discussions may be 
dominated by one or two participants, CMC can be inherently democratic allowing participants to participate 
equally in conversations (Herring & Stoerger, 2013). Furthermore, the Internet enables anonymity it 
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provides a forum where it is possible to be less inhibited and intimidated (Christopherson, 2007). Joinson 
(2001) defines this as the on-line disinhibition effect because without having to deal with f2f encounters, 
people can express themselves more openly. Thus, the socially anxious consider CMC more appropriate than 
f2f communication in potentially embarrassing situations because it may also reduce anxieties around 
negative evaluation from others (Yen et al; 2012). 

Critics of CMC argue the absence of non-verbal cues hinders communication and makes CMC a less rich 
form of communication (Dreyfus, 2008). However, the absence of non-verbal cues is an emancipatory factor 
allowing clearer communication by encouraging participants to be their true selves when on-line 
(Georgakopoulou, 2011). Also due to the lack of social cues, such as age and physical appearance, in CMC 
the level of self-disclosure is higher than in f2f communication (Saunders & Chester, 2008). It would be 
imprudent to claim that communication in real life situations is the same as CMC but it could be plausibly 
argued that CMC may well fulfil the needs of those teachers and students who feel anxious about discussing 
sensitive topics in a traditional f2f classroom setting. 

The objective of this case study was to explore whether the integration of aVLE into an RSE class would 
support communication between teachers and students. VLEs have been defined as “a software system that 
combines a number of tools that are used to systematically deliver content on-line and facilitate the learning 
experience around that content “(Weller, 2007, p.5). They have acquired a significant relevance as support 
tools for learning (Melton, 2006) by enabling collaboration, and the inclusion of constructivist strategies of 
collaborative learning into the instructional environment (Escobar -Rodriguez & Monge-Lozano, 2012) and 
are among the most widely adopted technologies in education (Risquez et al; 2015). 


2. METHOD AND RESULTS 

A four week study with 3 teachers and 28students (16 male, 12 females; average age 16 years) participating 
in an RSE class was conducted. A Moodle VLE was used for the study and the participants used the site from 
home to cover topics in the RSE curriculum including friendship qualities, self-esteem and sexual behaviour 
with educational activities (quizzes, videos, podcasts and fact sheets) designed to prompt discussions To 
encourage communication the users could engage in dialogue through email, a chat and a discussion board 
and could communicate anonymously using computer generated names. Anonymity is particularly beneficial 
for users discussing sensitive personal topics because it allows them to freely express their opinions and 
feelings in a confidential setting. Participants were encouraged to communicate on the site to discuss any 
issues relating to RSE that they felt was relevant to them. The 3 RSE teachers also used the VLE to respond 
to questions posed by the students and offer support where needed. In addition, the study presented the 
students with several tasks relating to RSE issues. Each week during the study the teacher posted a topic 
relating to the following week’s class. To support them, the students had access to suggested readings. These 
were provided in the form of short articles, typically consisting of one or two brief paragraphs relating to the 
topic. Quantitative and qualitative methods to address the research question were used to provide rich 
material for examination while supporting the trustworthiness of the findings through triangulation (Yin, 
2003). 

Most students 75% (f=l 1, m=10) reported they communicated more in the on-line RSE class compared to 
a traditional class. This finding was confirmed by teacher 3 who stated that “on/y about 5 or 6 of the class 
leaders would talk in class. Rarely would you get a whole class talking .” In addition, when comparing this 
result with statistical analysis from the on-line discussion forum (1857 posts in total and an overall average of 
22 posts), results suggest that a high level of interaction took place. Almost all the sample 86 % (f=9 m=15) 
reported they felt more comfortable discussing topics of a personal nature on-line than in the classroom. With 
respect to less feelings of embarrassment in the on-line environment, the overall result was 82 % (n=23) with 
36% (n=10) of females and 46% (n=13) of males reporting this. Several explanations may be surmised for 
this high score. Anonymity was perceived as the key factor by the majority of participants 82% (n= 23) for 
communicating more in the on-line environment, “It was easier to talk about stuff cos you’re not 
embarrassed and knowing that nobody could find out who you was great. ” “Sometimes it is embarrassing to 
ask things face to face. If I had to put my own name up there I don ’t think I would have been so open .’’ ’ 

79% (n=22) of the participants (f=12 m=10) reported that they felt able to talk about topics on-line that 
they would not normally talk about in class. A comparison with the teacher’s perception of the students’ 
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interactions elicited this comment from Teacher 1 “Its funny how you think you know people and yet on the 
site there are so different than they are in class. The people you think are shy in class have behaved 
differently on the site and able to use words that they wouldn ’t use in the classroom .” A comment from one 
of the students confirms this: “7 just wouldn ’t ask those questions in class and you don ’t really know if the 
teacher would answer the question or if the teacher would want to talk about those kinds of topics.” There 
was a difference among the teachers’ perceptions of the f2f interactions versus on-line interactions. Teacher 1 
noted, “7 like the fact that you didn ’t have to answer a question there and then. Sometimes you just don ’t 
know how to answer or if you do, you know that whatever you say will cause messing in the classroom, so I 
really liked that as it saved me a lot of embarrassment.” Teacher 3 found it easier to talk about topics on-line 
that she felt would cause embarrassment to both her and the students. “7 think the biggest fear as a teacher in 
a situation like that is being asked a question that you just don ’t know how to answer” For teacher 2, this was 
at odds with his usual pedagogy approach and questioned whether this may potentially damage his existing 
f2f rapport with pupils. He stated, “7 think I would miss the cut and thrust of conversation that goes on in a 
normal class”. However, it should be noted that this is at variance with the students’ perception of a 
traditional class. They reported that teachers did not welcome dialogue or debates in class. The narratives of 
the young people in this study imply that the common didactic style of teaching does not engage them or 
encourage them to ask questions. Comparisons with comments from two of the student interviews illustrate 

this difference: “Ye, in class, they just tell us about things but we can’t really ask them any 

questions ”. “In a normal class the teacher does most of the talking with little time for us to ask questions or 
discuss things that are important. ” 


3. DISCUSSION 

First, an important finding of this study is that development was driven by the needs and concerns of the 
students. Based on the students’ own accounts of their reasons for communicating they perceived the VLE as 
a space to express their concerns about topics that they wanted to know about but felt too inhibited to discuss 
in a traditional classroom. This supports Campbell & Aggleton’s, (2000) primary argument that to be 
effective sexuality education must meet the needs and interests of young people as conceptualized by them. 
Most students (82%) reported on the absolute significance of anonymity for communicating in a less 
inhibited way on-line. The reason for this becomes apparent when taking into account the research literature 
on the effects of anonymity (Joinson, 2001). Such a high reportage of less embarrassment (82%) may be 
attributed to what Suler (2004) refers to as the online disinhibition effect when people don’t have to worry 
how others respond to what they say with gestures such as a frown, a sigh or other signs of disapproval. This 
can be especially beneficial in an RSE class where individuals who find it difficult to communicate about 
sensitive topics can express their thoughts and concerns without fear of negative consequences. This may 
have important implications for RSE education as students are more likely to communicate more under these 
conditions. For many students the on-line class was a preferred method of communicating RSE to the 
traditional class. 75% (n=21) indicated that the on-line class should replace the traditional class. However, it 
is notable that more males (46%) than females (29%) preferred the on-line class. This may be due to gender 
differences for sources of information to learn and communicate in various domains (Turnbull et al; 2010). 
According to Rautopuro (2005) adolescent boys prefer to use computers to learn about sexual matters. 
Furthermore, while clearly the overall outcome is a positive response there are also two other possible 
explanations for this result. First, there is the possibility that the students’ appreciation was based on the 
novelty factor and therefore gave it increased attention and the second; students may simply view technology 
as commonplace and a natural tool to use. However, it may also be plausible that the positive reaction to the 
site was due to its similarity to the type of social networking sites that have become an integral component of 
their lives. Regarding the teacher’ experience, while the VLE was understood as a valuable communication 
tool, they tended to view the VLE as external to their regular educational workload. This was reflected in the 
difficulty they had in fitting the on-line activity into their already busy lives. On reflection, it may be 
inappropriate to look for a complete shift in the teachers’ fundamental approach to teaching and highlights 
the challenge of creating a truly innovative educational alternative within the context of a conventional 
school structure. Importantly, it also highlights that a balance is required between advantages for students 
and gains for the teaching staff in order for this method of teaching to be fully adopted. 
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There are several limitations to this study worth noting. The first is the use of a single institution for the 
data collected. This limitation may reduce the external validity of the results and somewhat limit the 
generalization of the findings. Second, it was only one relatively small study. Further work needs to be 
undertaken with a larger sample to provide additional evidence. 


REFERENCES 

Campbell, C. and Aggleton, P. 1999. Young People’s Sexual Health: A Framework for Policy Debate. The Canadian 
Journal of Human Sexuality, Vol. 8, No. 4, pp. 249-264. 

Christopherson, K.M. 2007. The positive and negative implications of anonymity in Internet social interactions: “On the 
Internet, nobody knows you’re a dog”. Computers in Human Behavior, 23, pp. 3038-3056. 

Dreyfus, H. L. 2002. Anonymity versus Commitment: the dangers of education on the Internet. Educational Philosophy 
and Theory, 34, pp. 369-378. 

Escobar-Rodriguez, T. and Monge-Lozano, P. 2012. The acceptance of Moodle technology by business administration 
students. Computers & Education, Vol. 58, 4, pp. 1085- 1093. 

Georgakopoulou, A. 2011. Computer-mediated communication. In: Ostman, J.O and Verschueren, J.eds. Pragmatics in 
Practice. Amsterdam: John Benjamins Publishing Co. pp. 93-108. 

Herring, S. and Stoerger, S. 2013. Gender and (A)nonymity in computer -mediated communication. In J.Holmes, M. 
Meyerhoff & S. Ehrlich (eds.). Handbook of Language and Gender. Hoboken, NJ:Wiley-Blackwell Publishing, 
pp.567 

Hyde, A. and Howlett E. 2004. Understanding Teenage Sexuality in Ireland. Dublin: Crisis Pregnancy Agency. 

Joinson, A. 2001. Self-disclosure in computer-mediated communication: The role of self-awareness and visual 
anonymity. European Journal of Social Psychology, Vol. 31, No. 2, pp. 177-192. 

Kanuga, M. and Rosenfeld, D. 2004. Adolescent Sexuality and the Internet: The Good, the Bad and the URL. Journal of 
Pediatric and Adolescent Gynecology, Vol. 17, No.2, pp. 117-124. 

Lee, C. S. 2010. Managing perceived communication failures with affordances of ICTs. Computers in Human Behavior, 
26, pp. 572-580. 

Lester, C. & Allan, A. 2006. Teenage sexual health needs: asking the consumers. Health Education, Vol. 106, No. 4, pp. 
315-328. 

Luppicini, R. 2007. Review of computer mediated communication research for education. Instructional Science, Vol. 35, 
No.2, pp. 141-185. 

Mannix-McNamara, P. and Geary, T. 2003. Implementation of Social, Personal and Health Education at Junior Cycle. 
National Survey Report. Dublin : Department of Education and Science. 

Mayock, P. et al; 2007. Relationships and Sexuality Education (RSE) in the context of social, Personal and Health 
Education (SPHE). Dublin: Crisis Pregnancy Agency. 

Rautopuro, J. et al; 2005. Students’ selection between virtual and traditional exam: factors explain the preferences to 
choose the study mode. European Conference on Educational Research, 7-10 September, Dublin 

Risquez, A. et al; 2015. The open dataset on students’ perceptions of virtual learning environments in Ireland: 
Collaborating to listen to the student voice. British Journal of Educational Technology, Vol. 46, 5, pp. 1070-1074. 

Saunders, P. L. and Chester, A. 2008. Shyness and the internet: Social problem or panacea? Computers in Human 
Behavior, Vol. 24, No.6, pp. 2649-2658. 

Selwyn, N. and Powell, E. 2007. Sex and relationships education in schools: the views and experiences of young people. 
Health Education, Vol. 107, No.2, pp.219-231. 

Turnbull, T. et al; 2010. Adolescents’ preferences regarding sex education and relationship education. Health Education 
Journal, 69, 3, pp. 277-286. 

Weller, M. 2007. Virtual Learning Environments: Using, choosing and developing your VLE. Oxon: New York: 
Routledge. 

World Health Organisation Regional Office for Europe. 2006. Sexuality Education in Europe - A Reference Guide to 
Policies and Practices. London: IPPF. 

World Health Organisation Regional Office for Europe .2008. Inequalities in young people’s health: Key findings from 
the health behaviour in School-aged Children. London: IPPF. 

Yen, J. et al; 2012. Social anxiety in online and real-life interaction and their associated factors. Cyberpsychology, 
Behavior, and Social Networking, Vol, 15, No. 1, pp. 7-12 

Yin, R. 2003. Case Study Research: Design and Methods. London: Sage Publications. 


256 


12th International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2015) 


THE DEVELOPMENT OF COMPUTATIONAL THINKING 
IN THE CONTEXT OF SCIENCE AND ENGINEERING 
PRACTICES: A SELF-REGULATED LEARNING 

APPROACH 


Erin E. Peters-Burton 1 , Timothy J. Cleary 2 and Anastasia Kitsantas 1 

‘George Mason University , College of Education and Human Development, Fairfax, VA 22030, USA 
2 Rutgers , The State University of New Jersey 
Graduate School of Applied and Professional Psychology, Piscataway, NJ 08854, USA 


ABSTRACT 

A quality educational experience for secondary students involves more than an acquisition of content knowledge; it 
entails providing students opportunities to develop a variety of thinking skills that enable integration of knowledge and 
the promotion of student self-directed learning outside of the classroom. One critical skill that is often underemphasized 
in education is computational thinking. The purpose of this conceptual paper is to discuss the parallels between the 
processes of computational thinking and self-regulated learning, and the corresponding implications of this integrated 
framework for instruction in secondary classrooms. Guiding our analysis is the premise that because computational 
thinking processes can be viewed as goal-directed processes, it is possible to use self-regulated learning theory as a 
framework for assessing and enhancing computational thinking. Secondary educators have minimal experience with 
teaching computational thinking in the United States, so not only is a clear definition of computational thinking necessary 
in the Next Generation Science Standards, it is also necessary to have a learning theory from which to structure this type 
of thinking. 


KEYWORDS 

Computational thinking; Computational practices; Self-regulated learning; Science and engineering practices 


1. INTRODUCTION 

Teaching content knowledge to secondary classrooms has been the focus of the educational system in the 
United States for many years, as seen in various national curriculum efforts (AAAS, 1993; NRC, 1996). 
However, a quality educational experience for secondary students involves more than an acquisition of 
content knowledge; it entails providing students opportunities to develop a variety of thinking skills that 
enable integration of knowledge and the promotion of student self-directed learning outside of the classroom. 
One avenue of thinking skills that is worthwhile to pursue is a literacy of the digital world because it allows 
the development of self-awareness of the symbolic nature of our world and the processes of one’s own 
learning otherwise known as metacognition (Wheeler, 2012). Along this line, a key thinking skill that is 
underused in the United States is computational thinking (CT), which is broadly defined as a way of “solving 
problems, designing systems and understanding human behavior by drawing on the concepts fundamental to 
computer science” (Wing, 2006, p. 33). Since few U.S. educators have much experience with CT, it is 
important to not only clearly define this term but to also examine and conceptualize this type of thinking 
from the lens of learning theories. Further, because it is important for students to be able to initiate and guide 
their use of CT, we believe that self-regulated learning (SRL) theories provide an ideal framework from 
which to explore how students engage in the computational thinking process (English & Kitsantas, 2013; 
Zimmerman, 1990). The purpose of this conceptual paper is to discuss the parallels between the processes of 
CT and SRL, and the corresponding implications of this integrated framework for instruction in secondary 
classrooms. Guiding our analysis is the premise that because CT processes can be viewed as regulatory, 
goal-directed processes, it is possible to use SRL theory as a framework for assessing and enhancing CT. 
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2. OVERVIEW OF COMPUTATIONAL THINKING 

There are three dimensions to computational thinking: computational concepts, computational practices, and 
computational perspectives. Computational concepts are the notions students use as they engage in design, 
such as iteration and parallelism; computational practices refers to the processes students develop as they 
engage with the concepts, such as debugging; and computational perspectives which includes the learners’ 
formation of viewpoints about the world and themselves. For the purposes of this paper, we are interested in 
the component of computational practices because they represent goal-directed behaviors and skills that 
comprise processes key to computational thinking. Such skills, defined in Table 1 (Grover & Pea, 2013), 
include the consideration of the benefits and limitations of the resources in an environment, reformulation of 
a difficult problem into one we can solve, and recursive thinking about correctness, efficiency, and aesthetics 
(Wing, 2006). 


Table 1. Computational Practices and Description 


Computational practices 

Description 

Abstractions and pattern generalizations 

Examining a group of patterns and describing them in a way that 
is clear and efficient 

Systematic processing of information 

Using heuristics to make sense of an event 

Symbol systems and representations 

Portraying an often abstract event with a simplified concrete 
object 

Algorithmic notions of flow control 

Managing data using a specified procedure 

Structured problem decomposition 

Breaking down a complex problem or system into parts that are 
easier to understand 

Iterative, recursive, and parallel thinking 

Repeating thinking in cycles to meet a goal (iterative), think 
about your thinking (recursive), and ability to focus thinking in 
specific directions (parallel) 

Conditional logic 

Asserting that the occurrence of one event depends on the 
occurrence of another 

Efficiency and performance constraints 

Considering hindering and beneficial factors involved in a 
process 

Debugging and systematic error detection 

Engaging in a methodical process for finding and reducing 
defects 


Because novices are often unaware of the many tactics embedded in computational practices (CPs) and 
will also struggle to use the strategies that they learn in class to solve problems, a framework is needed to 
delineate the process through which students engage in CT as well as their skill in managing the quality of 
this type of thinking. Much like any complex problem-solving activity, CT entails an integration of processes 
such as abstraction, decomposition, separation of concerns, problem solving, using randomization, designing 
systems, transforming, and simulating solutions. For novices, using these processes in an integrative fashion 
can be quite overwhelming. Therefore, we advocate conceptualizing CT as a series of goal-directed 
regulatory processes because it guide instruction and help students become more strategic and self-directed in 
their explorations using CT. Further, we believe that teachers can help students become “regulated” 
computational thinkers by providing feedback to students about the processes they use when engaging in this 
process. 

2.1 Self-Regulated Learning Theory 

Self-regulated learning (SRL) is viewed by many theorists as a goal -directed process whereby a person 
identifies a problem, examines relevant data to inform or develop and solution plan, implements this strategic 
plan, and then evaluates the effectiveness of this plan in achieving one’s goal. From our perspective, CPs 
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parallels the regulatory process because it involves making a series of judgements toward a solution to a 
problem, deploying tactics to iteratively conceptualize problems, identifying patterns in data such as points of 
divergence and convergence, and troubleshooting problems during the process. Also similar to SRL, to be 
able to engage in the iterative processes of CT, students must be motivated to do so, possess knowledge of 
tactics to reach their goals, and be able to effectively monitor and evaluate how well they are proceeding to a 
problem solution. 

Self-regulatory methods of learning have been shown to be effective in aiding learners to explicitly 
analyze the skills or knowledge that is needed to achieve a particular goal in areas such as academic studying 
(Cleary & Zimmerman, 2004; Thomas & Rohwer, 1986), use of instructional media (Henderson, 1986), 
metacognitive engagement (Corno & Mandinach, 1983), athletics (Kitsantas, Zimmerman, & Cleary, 2000), 
writing revision (Zimmerman & Kitsantas, 2002), and scientific epistemologies (Peters & Kitsantas, 2010). 
Using the iterative self-regulation model as a lens for CT, the indistinct task of teaching and learning CPs can 
be made more concrete. 

Zimmerman (2000) captured this process in terms of a cyclical feedback loop consisting of three phases: 
forethought, performance, and self-reflection (see Figure 1 ). Although this figure highlights abstraction and 
pattern generalization, any of the CPs could be applicable. The forethought phase refers to influential 
processes, such as analyzing tasks and setting process-oriented goals, that precede efforts to act and set the 
stage for action. The performance phase includes processes that occur during learning, such as 
implementation of task strategies and metacognitive-monitoring. The self-reflection phase refers to the 
processes that occur after learning or performance, such as the use of standards to make self-judgments about 
the performance, that collectively influence how a person responds to their performance. Because students 
continuously cycle through the feedback loop during learning, when students enter successive iterations of 
the loop, they are better positioned to use forethought, performance, and self-reflection processes in future 
learning. 

2.2 Self-Regulated Learning: Creating Learning Environments that Foster 
Computational Thinking in the Classroom 

Although self-regulatory processes are internally driven, they can be guided and facilitated by mentors and 
appropriately- constructed learning environments (Zimmerman, 2000), thus allowing a teacher to serve as a 
model to teach how CT is performed in a directed way. Since CT can be seen as a skill set for students to be 
competent in a problem-solving process, SRL can be a helpful framework to help students develop and use 
the tactics they need to employ to think computationally. By using a SRL framework, teachers can help 
students to organize information and skills, formulate solutions to a defined problem, and reflect on their 
problem-solving to create a more optimal process in the next iteration of problem solving. SRL is a way to 
manage the conceptual thinking process; an algorithm which analyzes thinking processes for the most 
effective path from problem identification to solution generation. 

Teachers can help students become “regulated” computational thinkers by helping them to self-monitor 
their use and application of CT skills and by also providing them with additional feedback regarding the 
processes they use and need to refine when engaging in CT. Measures of SRL, such as SRL microanalysis 
(Cleary, 2011), can be used as both an instructional prompt and as an assessment tool of student progress. 
Unlike traditional questionnaires or rating scales, this structured interview assessment protocol is considered 
a type of event measure of SRL because it examines students’ regulatory processes (goal-setting, planning) as 
students engage in specific learning tasks or activities. Because this type of assessment generates qualitative 
and quantitative, process-oriented information about how students engage in a task, teachers can use this 
approach and the corresponding data it yields to generate “actionable feedback” for students. Thus, although 
we know what CT is, we do not yet fully understand how to employ CT or how to help teachers better 
understand how their students engage in CT. Given that SRL microanalytic questions reveal information 
about how students set goals metacognitively monitor their progress, evaluate their performance, and adapt 
their CT, this approach can be extremely useful in obtaining information about students’ SRL processes 
while engaged in CPs and in providing information about students’ quality of execution of CPs for teacher 
assessment of student growth. 
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Performance phase 
Learner monitors progress of 
abstracting patterns, uses 
strategies to recognize 
similarities across patterns, and 
efficiently describes patterns 



Forethought phase 

Learner considers prior experiences with 
pattern recognition and modeling 
Learner recognizes how much they 
values the process, and sets goals to be 
successful in the process 


Task: Abstraction and 
Pattern Generalization 



Self-reflection phase 
Learner checks outcomes for 
accuracy of abstractions and clarity 
and efficiency in the generalizations 
described 

V J 


Figure 1. Self-regulated learning in abstraction and pattern generalization (adapted from Zimmerman, 2000) 

2.3 Teaching Computational Thinking in the Context of Science and 
Engineering Practices 

The minimal emphasis on CT in secondary curriculum in the U.S. is most readily observed in the most recent 
national standards in the U.S., the Next Generation Science Standards (NGSS). The NGSS weaves together 
three strands of learning domains: Disciplinary Core Ideas, Cross Cutting Concepts, and Science and 
Engineering Practices. Disciplinary Core Ideas set the factual content standards that are the building blocks 
of the curriculum. The Science and Engineering Practices explain the disciplinary process skills that embody 
the development of science content knowledge, and the Cross Cutting Concepts represent broad themes that 
help to categorize the other standards into ideas that are essential to science. In this format, the Science and 
Engineering Practices are central to being able to learn about how content ideas are generated in the 
discipline of science. As one of the eight practices, NGSS defines “Using Mathematics and Computational 
Thinking” as using mathematics to accomplish investigations, perform analyses, and build complex models 
(NRC Framework, 2012). 

It is notable that NGSS is not aligned to the ways the field defines CT. In fact, it describes CT more in 
terms of using mathematics. Because these standards will guide all future curriculum development, it is 
alarming that there is very little resemblance to the ways that the research-base defines CT and the practice of 
CT as defined in the NGSS. More efforts will be needed to help educators understand the gaps in NGSS 
descriptions of CT relative to the descriptions used in the field of computer science. We believe that teaching 
CPs through an SRL framework is one way that CT can be strengthened in curriculum. Strategically 
approaching how students are engaged in the task requires knowing what key features of CT we want 
students to know, and CPs provide a tangible, goal-oriented process from which to instruct and assess student 
learning. 


3. CONCLUSION 

Not only is CT underused in the United States (Wing, 2006), CT representation in national standards is 
glossed over and misaligned with the research -based concept of CT. Building students CT is a powerful and 
necessary component of instruction because it represents a universally applicable skill set. Although CT 
provides important skills, it is not well-represented in curriculum offered the United States. Secondary 
educators have minimal experience with teaching CT, so not only is a clear definition of CT necessary in the 
standards, but a learning theory from which to structure this type of thinking is also of vast importance. SRL 
theory provides a compelling and integrative framework from which CT skills can be taught. Constructing a 
mental toolkit of strategies from various perspectives to solve problems and design systems results in people 
who are better thinkers and learners. 
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ABSTRACT 

We have explored educational methods for algorithmic thinking for novices and implemented a block programming 
editor and a simple learning management system. In this paper, we propose a program/algorithm complexity metric 
specified for novice learners. This metric is based on the variable usage in arithmetic and relational formulas in learner’s 
algorithms. To evaluate the applicability of this metric for novice education, we discuss the differences between three 
previous program complexity metrics and our proposed metric. 
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1. INTRODUCTION 

Not only in undergraduate schools but also in pre-university education, 21st century skills are some of the 
most important learning topics [1,2]. In 2009, an OECD report pointed out three ICT related competences for 
new millennium learners [3]. 21st century skills were one of them. 21st century skills are a framework 
containing some skills, abilities and knowledge related to ICT. Recently, Informatics/Computing is one of 
the main subjects in junior or senior high school [4]. In these subjects, algorithms and algorithmic thinking 
are key topics [5]. However, many tools do not have ways to control the learning process. Also, many tools 
do not have ways to evaluate and assess student's work. 

We proposed an educational method for algorithmic thinking in fundamental education for computer 
science course students in 2008 [6] and examined the effectiveness of our algorithmic thinking learning 
support system. This system uses only three flow structure elements: calculation, selection and repetition. 
Students describe the algorithm using these three elements. To assess learner’s understanding, we evaluated 
each learner’s algorithm in detail. As a result, we found eight types of errors in their algorithms. Based on 
this fact, we think that if an instructor can give an appropriate sub-task to a learner who has misunderstanding 
or confusion when creating an algorithm, they can revise their mental model or knowledge of algorithm 
creation. To fulfill this goal, a program/algorithm complexity metric is needed in this study. 

This paper describes a program/algorithm complexity metric for our algorithmic thinking course and also 
discusses some possibilities of our proposed metric as an assessment function for novice learners. 


2. OUR APPROACH 

We developed a tool called AT to describe algorithms for novice learners. AT can describe the algorithm 
with block programming. Learners can make ten types of blocks in AT. Three types of blocks (plan, 
conditional branching and loop) can contain other blocks. 
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We use three learning items in this class: calculation, conditional branching and loop. In this study, four 
types of problems are created using a combination of these learning items: calculation problem, conditional 
branching problem, loop problem and combination problem. The calculation problem uses only calculations. 
The conditional branching problem uses calculations and conditional branching. The loop problem uses 
calculations and loops. The combination problem uses all the learning items. In this study, the maximum 
number of lines in the algorithm is thirty. 

The purpose of this study is to explore educational methods for algorithmic thinking conceptual modeling 
for novices, so we think that it is important to make support problems. Support problems are different from 
problems for all students who attend our algorithmic thinking class. Support problems are problems that 
overcome students’ weak points. We want all students to understand the algorithms. Therefore, we identify 
errors in students’ answers and teach these points to the students. Then we need to judge their weak points 
and make support problems to overcome these weak points. To make support problems we need a metric. 

There have been proposed some metrics used to evaluate programs. For example, McCabe’s cyclomatic 
complexity metric [7], Halstead’s complexity metric [8] and Hakuta’s complexity metric [9]. We discuss the 
problems with these three metrics related to our learning tool. First, the three metrics do not take into account 
the way the variables are used in calculations and control statements. Second, they can only evaluate 
algorithms that are complete. Third, they are not suitable when the level of difficulty of the learning item 
changes. Therefore, we need a new metric to resolve these problems. 

In this paper, we write about two points. The first point is to introduce our proposed new metric. The 
second point is to describe the characteristics and differences between previous metrics and our metric. 


3. EDUCATIONAL PROGRAM COMPEXITY METRIC FOR NOVICE 
LEARNERS: FL 

We propose a new metric to solve the problems with the three previous metrics. In this section we describe 
the characteristics and evaluation method of our metric. 


3.1 Evaluation Method 


The metric proposed in this study is called “Educational program complexity metric for novice learners” and 
its abbreviated designation is FL. In this study, calculations are limited to dyadic operations. Therefore, the 
number of operands and arithmetic operators and the kinds of variables are decided in calculations. Also, 
conditional expressions in control statements are limited to two operands and one comparison operator. We 
take into account the way operands and operators are used in calculations and conditional expressions. We 
also consider nested control statements. 

Table. 1-3 shows the way we evaluate each learning item. Table. 1 shows the levels for calculations. There 
are six calculation levels defined according to the number of operands and the kind of variables. Table. 2 
shows the levels for conditional branching, while Table. 3 shows the levels for loops. There are two 
conditional branching levels and two loop levels defined according to the variable or number used in the 
comparison in the conditional expression. 


Table 1. Calculation levels 


CL 

Summary 

Examples 

1 

Only numbers 

a = 0 
a= 1 +2 

2 

Different variable 
and a number 

a = b 
a = b + 1 

3 

Same variable and 
a number 

a = a + 1 

4 

Two variables 

a = b + c 

5 

Same variable and 
another variable 

a = a + b 

6 

Same variables 

a = a + a 


Table 2. Conditional Branching levels 


CBL 

Summary 

Example 

1 

Compare with 
a number 

if ( a < 1 ) 

2 

Compare with 
a variable 

if ( a < b ) 


Table 3. Loop levels 


LL 

Summary 

Example 

1 

Compare with 
a number 

while ( a < 1 ) 

2 

Compare with 
a variable 

while ( a < b ) 
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The features of our method to evaluate an algorithm are shown here. First, a level is assigned to each line 
in the algorithm using Tablel-3. The levels are denoted by CL, CBL and LL. Second, if each learning item is 
not a line, the levels of each learning item are added. We call these sums Scl> Scbl and S LL . Third, the levels 
of the three learning items are added. Finally, if there is a nested control statement, their number is added to 
the three learning items’ levels. We name this value R EA . The upper parts of Figure 1 and Figure 2 show two 
sample algorithms and the values that result from FL. The left algorithm is a calculation problem and the 
right algorithm is a combination problem. The calculation problem is to calculate the BMI with two variables 
(hei and wei) as input numbers. The combination problem is to calculate the product of two variables (x and 
y) for values of x and y from 1 to 9. If the product is divisible by two, the product is output. Sample 
calculations of FL are shown for the two algorithms in the upper parts of Figure 1 and Figure 2 in the lower 
parts of both Figure 1 and Figure 2. From this result, the R EA of the combination problem is higher than the 
calculation problem, so the combination problem is more complex than the calculation problem. 


CL 1 
CL 1 
CL 1 


CL 6 
CL 4 


Scbl - 0 
S L1 = 0 
N ncs = 0 

Rea =13+0 + 0+ 0 = 13 


Calculation problem> 



Combination problem> 



CL 1 
CL 1 
CL 1 

LL 1 
CL 1 
LL 1 
CL 4 
CL 2 
CBL 1 


CL 3 


CL 3 


Scbl - 1 
S L l =1 + 1=2 


Rea=16 + 1+2+2=21 


Figure 1. Result of calculation Figure 2. Result of combination 

We take into account the way operands are used in formulas and conditional expressions in control 
statements. Because FL sets levels for each learning item, we can evaluate each line in an algorithm. 
Therefore, we can evaluate incomplete algorithms. Also, because FL can transcribe error algorithms, it can 
make support problems in this study. 


3.2 Comparison of Result between Previous Metrics and FL 


Table. 4 shows the characteristics of the three previous metrics and FL. There are two points of view and 
seven items. The two points of view are structural point of view and educational point of view. The structural 
point of view looks at the structure of algorithms. The educational point of view is concerned with support 
problems. 

Metrics that have the characteristics (a)-(g) have a ^ in the corresponding column in Table. 4. First, 
McCabe's cyclomatic complexity metric cannot evaluate algorithms from a structural point of view and 
educational point of view in this study. Second, from a structural point of view, Flalstead's complexity metric 
can evaluate operands and operators in an algorithm. Flowever, it cannot evaluate nested control statements. 
From an educational point of view, it is impossible to change algorithms, so (e)-(g) do not apply to this 
metric. Third, from a structural point of view, Flakuta's complexity metric can evaluate algorithms because 
there are measures that evaluate operands, operators and algorithm length. From an educational point of 
view, the metric also does not apply to (e)-(g). Finally, FL sets levels in detail according to operands and 
operators. Therefore FL can evaluate algorithms from a structural point of view. Also, by adding the number 
of nested control statements, FL can evaluate them. Therefore, FL can evaluate algorithms like previous 
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metrics. From an educational point of view, FL sets levels to evaluate each line and each learning item, so all 
items apply. Therefore, FL is the only metric that can be used to make support problems. 

Table 4. Characteristic of previous metrics and FL 


Evaluation points 

Previous metrics 

FL 

McCabe 

Halstead 

Hakuta 


(a) Uses the number of 
variable types 


✓ 

✓ 

✓ 


(b) Uses the number of 
variables and numbers 


✓ 

✓ 

✓ 

5 

f 

(c) Metric changes with 
program length" 


✓ 

✓ 

✓ 

5 

(d) Considers nested 
control statements: there 
are two types of algorithms 



✓ 

✓ 


(e)Designed for use on 
unfinisfied programs 




✓ 

T 

(f) Can evaluate program 
subsets: to make support 
problems 




✓ 

5 

(g) Can evaluate different 
learning item levels 




✓ 


The purpose of this study is to explore educational methods for algorithmic thinking conceptual modeling 
for novices. Therefore, it is necessary to make support problems. Support problems are made based on errors 
in students’ solutions, so it is necessary to change the levels of problems. From Table. 4, all items apply to 
FL. Also, four items apply to Flalstead's complexity metric from a structural point of view. Therefore, we use 
FL to make support problems and use Flalstead's complexity metric to choose support problems suitable for 
each student. 


4. CONCLUSION 

In this paper, we proposed a program complexity metric based on variable usage for algorithmic thinking 
education. The features and problems of three previous metrics were discussed. The main problems with 
previous metrics were the three metrics do not take into account the way the variables are used in calculations 
and control statements. To avoid these problems, we developed a new metric for novice learners’ algorithmic 
thinking education. The characteristic of our proposal is that each learning item can be evaluated. Moreover, 
unlike previous metrics, our metric can evaluate each line in an algorithm. Therefore, we decided to use FL 
to make support problems for novice learners who submit algorithms with errors. 

We think this metric is suitable for evaluating novices’ algorithms. However, we did not evaluate the 
effectiveness of our metric in real educational situations. In the future, we will make rules to make support 
problems. Using these rules, we will make support problems according to the error levels in students’ 
solutions. After giving the support problems to students, we will check whether students’ understanding 
improves. 
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ABSTRACT 

The increasing use of ICT devices, such as smartphones and tablets, needs development of properly software or apps to 
facilitate socio-educative life of citizens in smart cities: Adaptive educational resources, leisure and entertainment 
facilities or mobile payment services, among others. Undoubtedly, all that is opening a new age with more information 
and autonomy for each individual, but the point is if these apps are accessible for the whole population. And when we 
talk about accessibility in an App, we are not only considering if the user is able to switch it on or unfold it main menu, 
we consider interface aspects that can present difficulties for some users. 

This study analyzes accessibility and usability of 15 Apps for people with visual dysfunction, because this difficulty has 
the greatest influence on the effective and efficient use of them. Data are collected through a descriptive scale made by a 
deductive-inductive process on four categories with a wide use of Apps. Social and specifically social -networking 
category is the largest consider accessibility on their apps, being highly demand by users and cost-effective for 
companies. Other categories are evaluated as completely inaccessible by users. 


KEYWORDS 

Accessibility; APP’s; Usability; ICT; Digital inclusion; Software evaluation; Visual impairment. 


1. INTRODUCTION AND ESTATE OF THE TOPIC 

On the last decades, the internet is used by most people to exchange information, compare and search for data 
or as a single entertainment, being this procedures as something absolutely socially accepted. But although, 
the internet has been remarkably successful like ubiquitous and universal mass media, there are still a lot of 
aspects to work out. Dominigue (2011) states, it is necessary to go forward on new ideas to satisfy 
investigation and society, to accept new challenges and chances of the digital society. 

The blooming emergence and use of Information and Communication Technologies (ICTs) by people, 
especially smartphones and tablets, brings a need to development properly smart software or apps, to get the 
best achievements on these devices, which have been well received for making our everyday life easier. 

The creation of hundreds of mobile applications try to make easier the city life, from discovering 
available entertainment venues every single day, to finding out the petrol station with lowest price, to the 
roads conditions at any time time. It is a fact that ICTs are opening a new age, improving and facilitating the 
information our personal autonomy. 

But what is exactly an “app”? This word comes from “application”, defined by Gil and 
Rodrlguez-Porrero (2013), as the software application used in mobiles devices and .specially designed for 
mobiles phones and tablets, accessed by a touchscreen. You can downloaded free or paid for, from websites 
of companies of these devices, although one of the widest world used is by Company Google, called “Play 
Store”. Also, they have easy installation, automatic updating, small file sizes and minimum storage 
capacity on the hard disk, despite their quite intuitive interface. Therefore only in a few minutes, the users 
can try these apps, erase and look for another one more suited to their interest on different platforms or apps 
stores. They can look for several tools, according to personal needs and abilities, turning their phones in a 
useful and powerful tool to facilitate their daily life. 

Are these new technologies actually accessible to everybody? The concept of accessibility is a pretty 
broad term and covers different fields. Broadly, it could be defined as the capacity of objects or places of 
being useful for any user, regardless of their abilities. Alonso (2007) defines it as “the possibility of going 
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where it’s required or achieve the things we wish (...) in the context of disability the term acquires an 
aggressive touch when it refers to rights” (p.16) As Alonso explains, this process of removing barriers 
implies quite complex interventions and it involves a high cost many times, because they are only aimed to a 
little percentage of population. On this way, an app is really accessible when the user can interact with all 
applications elements, regardless a physical, psychological, sensory disability or any other disability. 

The application elements means all the application’s interface elements, including the overview when you 
start the application, keys of control and so on. 

And the app’s accessibility is not only switch on, open and access to the application main menu, it means 
an easy interface to facilitate surfing on all elements of app, without barriers for some users. Also, Gil y 
Rodriguez (2013) explain other points to be considered by apps’ designers, such as organization of 
application, popups texts, visual appearance, color or contrast, because all of them have a direct influence 
on the accessibility and usability of the application. 


2. MATERIAL AND METHODS 

This study analyzes accessibility and usability of 15 Apps (see table 1 ) by people with visual disability, taken 
by an intentional selection of a large number of available at Google Play (Android operating system), App 
Store (iOS operating system) and iTunes Store (iOS operating system) rankings of most used applications of 
year 2014. 


Table 1. Apps’ sample 


App Store (iOS) 

Google Play (Android) 

iTunes Store (iOS) 

Facebook Messenger 

Duolingo 

Facebook Messenger 

Snapchat 

Facebook 

WhatsApp Messenger 

YouTube 

MyFitnessPal 

Telegram Messenger 

Facebook 

Netflix 

Facebook 

Instagram 

Pandora 

YouTube 

Pandora Radio 

NFL Mobile 

Google Maps 

Google Maps 

Flipagram 

Instagram 

Flipagram 

TripAdvisor 

Spotify Music 

Spotify Music 

Candy Crush Saga 

Shazam 

2048 


Wallapop 


The apps are grouped according to their environment of use: Educational environment (Google and 
Dropbox and Duolingo); Social environment (Facebook, Twitter, Youtube, Wallapop, EsAccesible and 
Fever); Health and Sports environment (Endomondo and MyFitnessPal) and Travelling environment 
(TripAdvisor, Blablacar, Madrid Metro/Bus/Cercanlas and Google Maps). After their descriptive analysis, 
we can establish their accessible and usable level by these visual disabled citizen. 

This difficulty has the greatest influence on the effective and efficient use electronic devices. They appear 
because of malformations, diseases or injuries at the sight organ of hereditary origin, dominant- recessive, or 
linked to gender. Depend on its severity, causes low-vision or blindness (Lou, 2011). The World Health 
Organization (WHO) (2014) divides vision ability in 4 levels in its briefing note 282: Normal vision, 
moderate visual disability, severe visual disability and blindness. Gathering the third from last and the second 
to last under “low-vision” that represents visual disability cases together with the blindness term. We can find 
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285 million people suffering from visual disability all around the world, 30 million blind and 246 low-vision 
rated. In light of this data, a large part of world population suffers with this disability, so thinking and look 
after these people is an obligation when it comes to the time of creating and designing open spaces, buildings 
and technologies, among which we can find the Apps. 

The descriptive scale, as instrument to register data, consider eighth indicators of accessibility. The 
following table 2, about education field, is showed as a single example one of the apps assessed on this field. 


Table 2. APP’s Accessibility Indicators on EDUCATION FIELD 


Indicators 

Google Chrome 

1. Images 
description 

1. Images, Graphics, 
colours and videos are not 
equally described. X 

2. Images, graphics, 
colours and videos are 
described in some 
situations. 

3. Images, graphics, 
colours and videos that 
appear are described. 

2. Media content 

1. It doesn’t let you make 
images, graphics nor 
videos big. 

2. It only lets you make 
content big in specific 
situations. 

3. It lets make all images, 
graphics and videos big. 
X 

3. Contrast, 
brightness and 
colour 

1 . Levels of colour, 
brightness and contrast 
are unchanging. X 

2. Only some of the 
levels can be changed and 
just in some situations. 

3. It allows to change 
colour, brightness and 
contrast levels. 

4. Auditory 
signals 

1. There’s no auditory 
signal that guides the 
process. X 

2. In some cases there are 
auditory signals. 

3. The app accompanies 
the information with 
auditory signals all the 
time. 

5. Touch surface 

1. The keys don’t have at 
least 9mm. 

2. Only some keys have 
at least 9mm. 

3. The keys have at least 
9 mm. X 

6. Customizable 
controls 

1. Interface doesn’t allow 
to modify controls on 
colour and size. X 

2. Some controls are 
alterable on colour and 
size. 

3. Every control is 
alterable on colour and 
size. 

7. Text 

1. It doesn’t let you make 
the text bigger. 

2. Some texts allow you 
to make its size bigger. 

3. Every text let’s you 
make its size bigger. X 

8. Clear messages 

1. Too much information. 
More than one phrase per 
informative act. 

2. Generally, information 
is specific but sometimes 
there’s too much text. 
More than one phrase per 
informative act. 

3. Information is specific 
and self-explanatory. It’s 
expressed with words, 
getting the message 
through. 

X 


3. OUTCOMES 

In the educational field this study consider the wide world uses browser Google, the cloud application 
Dropbox and the online languages course Duolingo. Although all of them are commonly used in the 
Educational context, they can be also easily included in other fields. 

Regarding their images description, none of them give any additional information and they just attach 
their contents. Something similar happens to increase size of media contents, except by Google Chrome, 
because its navigation function on the touchscreen to enable the size by a finger touch. 

Contrast, brightness and color are basic point to facilitate accessibility to a large group of people to 
these applications. Nevertheless, anyone of them lets user adjust it. That could be half justified because the 
control of these on the device’s control keys, although the app should make it easier. 

The beeps and sounds are really important to make a difference, for visual disable people, to be able to 
manage or not an application. Anyone of applications analyzed give this options. The opposite happens about 
a minimum touch surface, fulfilled by all of them. 
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Another essential point is capacity to increase size of letters, texts and messages. That is only facilitate 
by Google Chrome, due to its function, however the other apps do not it. 

Also, the three apps analysed in the educational field, show something common. That is the indicator 8 
about clear messages, because all the information displayed on the apps is clear and concise. 

As an overall outcomes on all environments analyzed, Apps can be organized in four categories, 
depending on the relation of citizen and users with them: 

• Full Accessible App. App interface facilitate adaptation of several points required by users 

• Partly Accessibility App. App does not facilitate accessible elements. So, users need special tools to 
adapt and transmit the content, such as Talk-Back. 

• Twin Accessible App. App created along the original for specials users to make up its lack of 
accessibility. 

• Non Accessible and Usable App. App interface cannot facilitate ant adaptation for users who requires it. 
This work concludes on two main ideas: Firstly, some accessible points all usual in all apps analyzed, 

such as a touch surface of 9 mm or clearance of information displayed, but most of the points about 
accessibility are not so good. And regards the accessibility issues, all points on the apps should be balanced 
integrated, if not a couple of positive points, do not count for rising this points. 

And. Apps used on a social field are widely businesses by software companies, trying to give users 
several possibilities to their demands, and therefore, these apps have usually more accessible points than 
others. Being important for instance, possibility of rising size of the texts or media contents. On this social 
field are located social networks, the most used by the world’s population, despite not being totally accessible 
most of them or having important points to work on. 

Summarizing, this work show shocking data about the ICTs in present days, widely used by everybody, 
with major investments by governments and private companies, and on which the citizen hopes for help to 
make theirs life a little easier. But yet, for part of the population with disabilities, the digital divide is getting 
wider every day. 

Considering that communication and information technology make everything easier on a daily basis for 
people. However the digital divide is growing every day. Disabled users can’t be an active part in their 
society, that separates them and makes them be just passive agents. 
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ABSTRACT 

Massive Online Open Courses (MOOCs) can create large scale communities of learners who collaborate, interact and 
discuss learning materials and activities. MOOCs are often delivered multiple times with similar content to different 
cohorts of learners. However, research into the differences of learner communication, behavior and expectation between 
multiple presentations is scarce. This is of importance to MOOC developers, academics and moderators, as an 
understanding of these differences could have an impact on content provision, community moderation, course delivery, 
learner interactions, and completion rates. This case study of two presentations of a Futureleam History MOOC examined 
learner activity data, and pre and post course learner survey results (n=10,449). Differences in learner survival rates, 
behavior, expectation, recruitment, experience of online learning, demographic makeup, reasons for non-completion, and 
comment activity were identified. These results form a preliminary exploration of learner community differences between 
multiple MOOC presentations that guide future analyses by identifying areas of comparative interest and importance. 


KEYWORDS 

Massive Online Open Course, Online Learners, Online Education, Learner Community Behavior 


1. INTRODUCTION 

Massive Open Online Courses (MOOCs) are a relatively new learning phenomenon where learners access 
freely available online educational multimedia resources online, and connect with large numbers of other 
learners asynchronously via social networking tools such as discussion forums (Liyanagunawardena et al., 
2013). These forums are used to facilitate course understanding, support learner problem solving, and build a 
community of learners. MOOCs are often repeated in multiple presentations with new cohorts of learners. 
Exploring and identifying where learner and community behavior is different between these instances can 
help MOOC moderators, academics and administrators in their development, implementation and evaluation 
of MOOCs. This paper provides a comparative analysis of learner behavior and demographics in two 
presentations of the same MOOC. It contributes an understanding of differences in learner community 
behavior across multiple presentations of a MOOC, and forms a preliminary basis for further work within 
this space. 

Although there have been comparative analyses across different MOOCs, comparative research is lacking 
on learner behavior, communication, and demographics within repeated presentations of the same MOOC. 
This could have an impact on MOOC content design, resource provision, community moderation, course 
delivery, learner / educator interactions, and completion rates. Comparative research across multiple MOOCs 
has explored learner activity rates and peer interaction (Hew, 2015), forum activity (Brinton et al., 2013), and 
pedagogical practices (Toven-Lindsey et al., 2015). However, little research has been conducted into 
repeated iterations of the same MOOC. Previous research into repeated versions of a MOOC has explored 
demographic variables (Liyanagunawardena et al., 2013), and has found that participation initially declines 
by 43% between the first to the second MOOC presentation (Ho et al., 2014). However, little comparative 
information as to community and learner activity, communication, survival and behavior, with a focus on 
multiple presentations of the same MOOC is available. MOOC providers often have data across multiple 
MOOC presentations; however, at present it is unknown which comparative variables are of importance, and 
what are the implications of differing MOOC learner populations. 
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2. RESEARCH AIM 

This research aims to examine differences in learner communication, behavior, demographics and survival 
across multiple presentations of a MOOC; the inaugural presentation, and the second presentation. It 
questions whether different cohorts of MOOC learners behave differently, and in what way. Identifying 
variables for comparison across MOOC presentations could help future research and guide content provision, 
community moderation, course delivery, learner / educator interactions, and completion rates. 


3. METHODOLOGY 

Two presentations of the ‘Irish Lives’ MOOC were delivered on the Futurelearn platform in September 2014 
(PI ) and March 2015 (P2) by Trinity College Dublin, Ireland. This was a six week Irish History MOOC, with 
four hours recommended study time per week. Assessment was based on two peer review essays, and 
completion of MOOC steps. A pre course and post course survey were administered to learners in both 
presentations. The pre course survey consisted of 20 questions related to demographic information, 
experience of online learning, expectations of online learning, and interest in history. The post course survey 
included 29 questions on course activity and satisfaction with online learning experience. Table 1 describes 
the sample of the pre and post course surveys, and the total number of joiners within both MOOC 
presentations (a joiner is defined as a learner course enrolment). Both presentations of the MOOC had the 
same content and moderation, except for one addition in the first step of P2 where learners were asked to 
introduce themselves. 

Table 1. Number of participants in pre course and post course survey responses and response rates 

MOOC presentation Total Number of Joiners Pre course survey participants Post course survey participants 

PI 18,264 3,834(20.9%) 1,722(9.4%) 

P2 11,085 4,458 (40.3%) 435(3.9%) 


4. RESULTS 


The results of the comparative analysis of multiple presentations of the Irish Lives MOOC suggest 
differences in learner survival, expectation, recruitment, experience of online learning, demographic makeup 
(age and gender), and educational level. Additional variables that suggest difference in MOOC presentations 
include reasons for non-completion, learner activity types, and comment activity. 


4.1 Learner Demographics, Experience, Expectation, and Recruitment 


The pre-course survey described significant differences between the two presentations in gender, age, 
educational level and experience in online learning. Table 2 describes how learners from PI were older, had a 
lower educational level, and the majority were geographically based in Ireland. A greater percentage of 
learners were female and from the United States in the P2. It also describes that PI had less experience of 
online learning than P2. 


Table 2. Demographic Information and Experience of Online Learning 


Gender* 

PI 

P2 

Male 

1,605 (41.9%) 

1,608(36.1%) 

Female 

2,030 (52.9%) 

2,656 (59.6%) 

Other 

Age** 

4(0.1%) 

8 (0.2%) 

0-35 

652(17.8%) 

858(20.1%) 

36-55 

1,324(36.2%) 

1,552(36.3%) 

56+ 

1,680 (45%) 

1,869(43.7%) 
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Educational Level* 

Primary 

110(3.1%) 

75 0.8%) 

Secondary 

950 (26.6%) 

1035(24.8%) 

Third Level 

2,512(70.3%) 

3,061 (73.4%) 

Geographical Location 

Ireland 

1,679 (46.2%) 

975 (22.7%) 

US 

844 (23.2%) 

1,754(40.9%) 

UK 

652(17.9%) 

861 (20.1%) 

Other 

455 02.5%) 

691 (16.1%) 

Experience of Online Learning* 

Yes 

1,045 (27.2%) 

1,559(35%) 

No 

2,592 (67.6%) 

2,736 (61.4%) 


*p<0.001 **p<0.01 

Significant differences in learner expectation also emerged in the responses ‘learn new things’, ‘try out 
learning online’, and ‘supplement my existing studies’. PI learners expected to try out learning outline to a 
greater extent than P2 learners, whereas P2 learners had a greater focus on career and study development. 
There were observed differences in how learners were recruited in PI than P2. Learners from P2 were 
recruited to a greater extent from online sources, whereas PI learners were recruited from print media and 
word of mouth. The post-course survey results found that there were significant differences in learner 
dropout rationale between the two MOOC presentations. These differences were apparent in the responses 
‘didn’t have enough spare time’ and ‘couldn’t keep up with the course’, with PI having significantly higher 
responses to these questions (p>0.01 ). Significant differences were also identified in ‘satisfaction with learner 
activity’. P2 learners enjoyed discussing content with other learners more than PI. 


4.2 Learner Survival 


Activity data extracted from the Futurelearn website found that there was a much higher dropout of students 
in the first day of the MOOC in PI than P2. Figure 1 describes the day of the last step that a learner was 
active within. The majority of PI dropouts were in the first day of the MOOC compared with the end of the 
first week in P2. This finding suggests that the behavior of learners in the first presentation of a MOOC can 
differ from the second. In effect, PI learners tend to drop out earlier than P2 learners. 
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Days tram launch 


Figure 1. Comparing last step progress in PI and P2 


4.3 Learner Behavior 


Comparing the number of learner comments in each step of the MOOC identified some interesting results 
(see Figure 2). Although most of the results are very similar, a spike can be seen in the third step of the first 
week in P2. This was due to the inclusion of additional text asking the learners to introduce themselves that 
was not in PI. This finding demonstrates how a small change in MOOC content across multiple 
presentations, can drive changes in learner comment behavior. 
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Figure 2. Comparing the number of comments in each step of the MOOC between PI and P2 

Learner behavior also differed between the two presentations, in particular between social learners (a 
learner is one who has made at least one comment) and participating learners (a learner who has completed 
the majority of the steps). PI had greater numbers of social learners (41%) and participating learners (25%) 
than P2 (36%; 16%). 


5. DISCUSSION AND CONCLUSION 

This research suggests that MOOC iterations can differ in learner expectation, learner recruitment, 
experience of online learning, demographic makeup, reasons for non-completion, learner activity types, 
comment activity and learner survival. Identifying these variables could have implications for MOOC design, 
implementation and evaluation. The practical implications of these results suggest that pre course surveys 
could be analyzed prior to the commencement of the MOOC to guide MOOC moderation. Post course 
surveys and MOOC data analysis can identify areas of importance for future MOOC iterations. We are 
curious to discover whether similar findings of difference will be apparent using data from other MOOCs. 
The following research questions have emerged from this analysis that warrant these future investigations: 

• Are learners from inaugural presentations more motivated to complete or comment on a MOOC 
than subsequent presentations (i.e. social and participating learners)? 

• Are inaugural learners more likely to drop out earlier than second presentation learners, and if so, 
why? 

• Does experience of online learning differ across multiple presentations, and if so, why? 

• Should MOOCs be compared across their presentation iteration rather than as an aggregate (i.e. first 
presentations be compared with first presentations, second with second)? 

• Are there patterns of demographic differences across MOOC presentations, and if so why? 

• Does learner activity and satisfaction with learning activities differ between presentations? 

This research described differences in demographic composition, survival, communication, activity, and 
behavior between MOOC presentations. The research is limited to one case study, however future research 
will examine these questions in additional MOOCs. This could determine whether there are common 
behavioral differences in MOOC presentations which could have an impact on their design and 
implementation. 
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ABSTRACT 

This paper explores the use of a three dimensional virtual world (3-DVW) to delivery assertiveness training to young 
adolescents. The case study aims to understand how a sense of presence in VWs facilitates and affect the performance of 
students role-playing activities to enhance their ability to resist sexual coercion. The results indicate that a sense of 
presence had a modest influence on participants’ verbal interactions while role-playing. 
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1. INTRODUCTION 

Adolescence (10-19 years) is a period of development marked by the emergence of romantic relationships. 
Over 20% of all adolescents have experienced either psychological or physical coercion from an intimate 
partner (Centers for Disease Control and Prevention, 2012). When adolescents lack belief in their efficacy to 
assertively express wishes or initiate discussions around safer sex, they are likely to be coerced in their sexual 
relationships and be at risk of negative outcomes of sexual behaviour such as HIV/ STIs and unintended 
pregnancies (Wolfe et al; 2012). Individuals who have experienced sexually coercive behaviour experience 
considerable traumatic health consequences such as physical and mental health problems, substance misuse, 
self-harm, antisocial behavior, and sexual behavior problems (Lehrer et al;2013). Victims may also feel 
violated, vulnerable, confused, betrayed, guilt and shame (DeGue & DiLillo, 2005). Young and Furman 
(2008) reported that after an early incident of sexual coercion, adolescents’ risk for subsequent incidents 
increased more than sevenfold. Programs focus on reducing vulnerability to sexual coercion by training 
people on how to protect themselves. Self-protection strategies including communication components aimed 
at improving the ability of participants to communicate assertively have been found to have some success 
(Gidycz et al; 2001 ). 

Role-playing is an approach to help individuals develop strategies for dealing with complex social 
situations (Jouriles et al; 2009). However, it typically involves learning the skills in a classroom setting 
resulting in an unrealistic and disengaging experience (Sognure, 2004). Students find the scrutiny of the 
group confronting, and performing in front of their peers frightening and embarrassing (Arnold & Koczwara, 
2006). Consequently, behavioral inhibition can impact on the content, style and quality of verbal interactions 
between individuals practicing role-plays (Hamilton et al; 2014). 

1.1 3D Virtual Worlds (3D VWs): Virtual Presence and Virtual Realism 

3DVWs are computer generated spaces with affordances for communication activities (Mennecke et al; 
201 1 ). Avatars represent the user in the VW and it is the user avatar that actually interacts with the world and 
other users via a range of synchronous and asynchronous verbal communication channels. These 
communication and exchange mechanisms in VWs create a sense of ‘being there’ at the same time and in a 
‘place’ with other avatars (Minocha, 2010). 
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Presence, the subjective sense of being in a place, is a defining feature of the learners’ experience that 
mediates the nature of communication in VWs (Mennecke et al; 2011). It is encouraged by the layers of 
sensory information which enable users to transcend the real world to see, hear and feel as if they are actually 
inside these environments (Blascovich & Bailenson, 2011). Presence is the combination of the technical 
capabilities of the medium, which encourages user immersion, and the level of psychological sense an 
individual has of being in the VW. Presence affects verbal interactions in role-plays. For instance, students 
in VWs take additional turns in conversations and have shorter exchanges in each turn than those in the 
Face-to- Face (Gao et al; 2008). Also, a sense of presence contributes to the argumentative knowledge 
construction by participants while role-playing in Second Life (Jamaludin et al; 2009). 

The extent to which the realism of the environment enhances role-play interaction has been contested by 
arguing that imitating reality is not particularly helpful to learning (Dillenbourg; 2008). However, the context 
of the learning is relevant particularly in the case of role-play pedagogy. For instance a program to reduce 
sexual victimization among adolescent girls, which used virtual reality headsets, found the medium provided 
a realistic setting for participants to practice the skills needed to resist sexual coercion since it allowed them 
to experience a heightened sense of anxiety similar to how they would feel in a real-life sexually coercive 
situation (Rowe et al; 2015). Similarly, the graphical realism of VWs has been successfully applied to 
practice role-plays in social situations that individuals may find difficult in the real world such as job 
interview simulations (Bell & Weinstein, 2011) and teacher training (Gregory & Masters, 2012). 
Notwithstanding the previous, limitations of VWs such as lack of visual features: facial expressions and 
subtle movements; may detract from a sense of presence and hence reduced verbal interaction and 
engagement. Nonetheless, Schroder (2002) argues users are able to cope well with the absence of certain 
features in the real world or face -to- face. Furthermore, it can be anticipated that advances in technology will 
generate progress in higher avatar realism to allow the transmission of similar social cues of face-to-face 
communication. 

This exploratory case study aims to develop a more in-depth understanding of how physical presence and 
sense of realism in a VW is perceived by students to influence their verbal performance while enacting 
role-plays. 


2. METHOD AND RESULTS 

17 students (10 male and 7 female) aged 13-14 years, attending a Social Health and Personal Education class 
participated in the study. None had experience of using Second Life and two students had experience of 
role-playing. Prior to initiating the study ethical research clearance was gained. 

Participants completed a four week assertiveness training course with two components: a) Theory: 
covering the concepts of assertiveness, aggressiveness and passiveness; and b) Practical: focused on 
role-plays to teach assertive strategies to resist pressure to engage in negative sexual behaviours. Assertive 
strategies covered included: assertive responses (speaking clearly, honestly and confidently, saying no, 
making requests and dealing with aggression). To support the students, an actor of similar age to the 
participants assisted by demonstrating the skills. Initially, students were provided with eight partially scripted 
role-plays and asked to respond with what they would say in a similar situation using the communication 
strategies they observed. Skill prompts in the form of sentence starters were also provided. On completion of 
the assertiveness course the students participated in the VW component of the study. 

The VW sessions took place in a computer lab over 6 weeks and each session lasted 40 minutes. During 
the first session, the students learnt how to navigate the VW and customised the appearance of their avatars 
as research indicates identifying closely with one’s avatar is a significant element in how an individual 
interacts with others (Ducheneaut et al; 2009). Once acquainted with the VW, participants role-played the 
assertiveness techniques learnt, in pairs or groups of three or four. For instance, a scenario for two required 
students to assertively refuse to kiss somebody they had just met at a party, while a group activity asked 
participants to convince an individual to send an explicit image of themselves on their mobile phone. The 
students took turns being the persuader and the person being persuaded. The aim of these scenarios was to 
practice assertive refusal skills which refer to the ability to effectively communicate through verbal (clarity of 
expression, tone of voice, inflection and volume, response latency) and non-verbal (body orientation, posture 
and gestures) an unwillingness to engage in negative sexual practices. 
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After the VW sessions, the students completed a survey exploring the participants’ engagement with the 
role-plays while in the VW. Users’ experience of presence and realism while in the VW was measured by a 
scales from Slater et al; ( 1998) and Witmer & Singer (1998) which required participants to rate statements on 
a five-point Likert scale regarding the participants’ perception of how the features of presence and sense of 
realism helped them enact the role-plays. 

Regarding the sense of Presence, the questionnaire results indicated that (n=6) students felt Much 
physically present and a similar number Somewhat physically present, while (n=3) students reported they felt 
A Little and (n=2) responded Not Much. Concerning how a sense of presence affected their verbal 
interactions during the role-plays, the numbers were similar. This would be expected as those who reported 
they did not feel physically present or stated A little would not perceive the environment affected them in the 
same way as those who felt physically present. However, from our data, it is not possible to discern whether 
the complete lack of physical presence reported by two participants occurred due to a lack of interest in 
learning in the VW or to the VW itself. While no student reported feeling Very Much physically present, 
(n=2) felt physical presence helped them to speak Very Much with confidence, indicating that perhaps it is 
not necessary to feel completely present in the environment. Regarding how presence helped the students 
express themselves, (n=4) students indicating Very Much, five Much, (n=5) Somewhat, (n=2) A little and 
(n=l ) student indicating Not Much. Being in the VW helped several of the students (n=7) to respond quicker 
during the role-plays. This may be due to several factors: students could practice as often as they felt 
necessary having plenty of time to work out how to respond during the role-plays; also, face-to-face 
discussions can present pressures on individuals to respond that perhaps can be managed better in a VW. 
Concerning how presence influenced tone of voice, (n=7) students reported a sense of presence helped them 
vary their tone of voice during the role-plays. Communication experts point to the role that anxiety can have 
on our speech patterns so it may be the case as has been suggested by Vallance et al (2014) that role-playing 
as an avatar reduces the anxiety experienced in face-to-face role-playing. However, the notion that the 
employment of avatars in role-plays influences an individual’s involvement in the interaction (Bailenson et 
al; 2006) may also be a contributing factor as speech patterns such as tone of voice are influenced by how 
engaged individuals are in the conversation. 

The findings of this study relating to the participants’ sense of realism while in the VW revealed that 
several students (n=8) felt the environment was Somewhat real, although this contrasted with the affect a 
sense of realism had on their verbal interaction. For instance, while (n=8) students found it Somewhat real 
(n=l) student found the sense of realism helped them speak Very Much with confidence. (n=6) students also 
felt the sense of realism contributed Much to speaking with confidence while (n=7) felt it Somewhat helped. 
A sense of realism did not have the same influence on tone of voice as reported with the same question in the 
presence section with only (n-2) students reporting Very Much compared to (n=7) students in the Presence 
section. Similarly, a sense of realism helped only (n=2) students use a moderate tone of voice compared to 
(n=6) students in the presence section. All of the students reported feeling engaged with the role plays to 
varying degrees with (n=6) students reporting they were Very Much engaged and (n=6) reporting that feeling 
engaged helped them express themselves well in the role-plays. 


3. CONCLUSION 

Teaching sexual coercion prevention strategies to adolescents requires innovative approaches. This study 
provides preliminary support for the efficacy of 3DVWs for role-play to teach assertiveness skills. We found 
that a sense of presence coupled with a sense of realism had a modest influence on participants’ verbal 
interactions while enacting the role-plays. While role-playing in VWs may not be better than face-to-face 
they are certainly comparable. This was an exploratory study with several limitations including a relatively 
small student cohort and reliance on self-reported data. In addition, while the current work is an initial insight 
into adolescents’ experience of VWs for teaching assertiveness skills, incorporating qualitative data in future 
studies will strengthen the evidence that a VW is an appropriate intervention for adolescents. 
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ABSTRACT 

Language may ordinarily account for difficulties in solving word problems and this is particularly true if mathematical 
word problems are taught in a language other than one’s native language. Research into cognitive load may offer a clear 
theoretical framework when investigating word problems because memory, specifically working memory, plays a major 
role in solving problems successfully. The main purpose of this study was to investigate the influence of language when 
solving mathematical word problems while taking into consideration participant’s limited working memory. The 
participants’ main role was to solve word problems in a format that depended on the group they were assigned to. The 
study utilized a qualitative method approach and involved three phases, a pre-testing, acquisition, and testing phase. 
Predominant findings from this study show that there was a statistically significant difference between the various groups 
participants were assigned to. 


KEYWORDS 

Cognitive load theory, mathematics, second language, and problem solving. 


1. INTRODUCTION 

The issues involving teaching college level mathematics in the United Arab Emirates (UAE) pose many 
challenges for teaching faculty and also highlight the need to thoroughly understand these challenges in an 
attempt to provide instructional guidelines that may help in enhancing student performance scores in 
mathematics. One of the most obvious challenges would seem to relate to teaching and learning mathematics 
in a second language. For instance, Emirati students wishing to join Zayed University (ZU) - a federal 
institution in the UAE- are required to enrol in several mathematics courses. Since the language of instruction 
is English, mathematic courses are taught in English despite the fact that for the majority of ZU students, 
English is a second language. To date performance scores have not been very promising and recent 
investigations have sought ways to improve the learning of mathematical word problems. 


2. LITERATURE REVIEW 
2.1 Cognitive Load Theory 

Cognitive load theory (CLT) may offer a theoretical framework for understanding how materials and the way 
in which they are presented may in fact impact learning. Cognitive load theory draws on the work of 
Atkinson and Shriffm (1968), which highlights a limited working memory that can be overloaded based on 
how information is presented. CLT research has identified a number of instructional designs, two of which 
are particularly relevant to this study: redundancy effect and modality effect (Sweller, 2004). The redundancy 
principle suggests that it is more effective to replace multiple information sources that can be understood on 
their own with just one source of information (see Moussa-Inaty, Ayres, & Sweller, 2012). The modality 
principle suggests that learning might be enhanced by replacing written explanatory text and another source 
of visual information with a spoken explanatory text and a visual source of information. 
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2.2 Mathematics and Language 

When solving mathematical word problems students not only need to know the correct mathematical 
operations in order to answer the tasks at hand, it is also essential for them to be able to successfully deal 
with the linguistic components - the lexis and syntax. The links between mathematics and language and 
indeed mathematics as a language have long been studied (see Halliday, 1978; Pimm, 1987). Lexical 
problems can be compounded by the fact that teachers in the UAE education sectors are from a wide variety 
of backgrounds (McKinnon, Moussa-Inaty, & Barza, 2014) bringing a number of first languages into the 
classroom. 


3. METHODOLOGY 

3.1 Purpose of this Study 

The general purpose of the study was to extend the research into the redundancy and the modality effect by 
specifically examining the impact of spoken and written text in English only vs. spoken and written text in 
English and Arabic when learning how to solve mathematical word problems. 

3.2 Participants 

A sample of one hundred and thirty two undergraduate female students from a federal institution agreed to 
take part in this study (see Table 1). Though the language of instruction was in English, the participants’ first 
language was Arabic, but they were all able to speak and understand English with an average IELTS score of 
4.6 for reading and 4.9 for listening, indicating that participants are moderate users of English. All 
participants were registered in a university college course (MTH 101) that mainly covers secondary 
mathematics and is specifically designed as a remedial class prior to taking the first year general mathematics 
courses. 


Table 1. Number of participants per treatment group 


n 

Groups Listening Only - English 18 

Listening Only - English & Arabic 20 

Reading Only - English 20 

Reading Only - English & Arabic 25 

Reading & Listening - English 26 

Reading & Listening - English & 23 

Arabic 


3.3 Materials and Procedure 

The experiment consisted of three phases, a pretest, an acquisition phase and a testing phase. For the pretest, 
a set of multiple-choice questions related to finding percentages were asked. For each activity in the 
acquisition phase both reading and auditory materials were required. For the testing phase, a combination of 
questions that tested for transfer was presented. To measure cognitive load, a 9-point likert scale was 
administered after a three phases of the experiment was complete. Students were assigned to one of six 
treatment groups, namely Read in English (RE), Read in English and Arabic (REA), Listen in English (LE), 
Listen in English and Arabic (LEA), Read and Listen in English (RLE), and Read and Listen in English and 
Arabic (RLEA). The format of the materials was presented based on the group the students were assigned to. 
The analyses focused on aspects of the problems presented that potentially increased cognitive demands for 
second-language learners. 
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4. RESULTS AND DISCUSSION 

Table 2 shows the mean rank for each treatment group only for those with significance below 0.05. 


Table 2. Mean Rank 



Treatment 

n 

Mean Rank 


Reading Only - English 

20 

63.53 


Reading Only - English and Arabic 

25 

69.62 

Acquisition 

3&4 

Listening Only - English 

18 

49.36 

Listening Only - English and Arabic 

20 

66.28 

Reading and Listening - English 

26 

57.77 


Reading and Listening - English and Arabic 

23 

89.17 


Total 

132 



Reading Only - English 

20 

62.08 


Reading Only - English and Arabic 

25 

69.14 

Acquisition 

Total 

Listening Only - English 

18 

47.97 

Listening Only - English and Arabic 

20 

66.93 

Reading and Listening - English 

26 

61.50 


Reading and Listening - English and Arabic 

23 

87.26 


Total 

132 



Reading Only - English 

19 

57.61 


Reading Only - English and Arabic 

25 

54.52 

Acquisition 

MER 

Listening Only - English 

18 

99.19 

Listening Only - English and Arabic 

19 

65.66 

Reading and Listening - English 

26 

69.62 


Reading and Listening - English and Arabic 

22 

49.30 


Total 

129 



Reading Only - English 

20 

77.70 


Reading Only - English and Arabic 

25 

65.20 

Post-Test 2 

Listening Only - English 

18 

69.31 

Total 

Listening Only - English and Arabic 

20 

58.75 

Reading and Listening - English 

26 

53.75 


Reading and Listening - English and Arabic 

23 

77.13 


Total 

132 



Kruskall-Wallis (KW) test results showed that there was a statistically significant difference between the 
distributions of Acquisition 3 & 4, Acquisition Total, Acquisition Mental Effort Ratings, and Post-Test 2 
Total. For each test with statistical significant differences, pairwise comparisons between treatments were 
conducted. For Acquisition 3&4, the results for RFEA was higher than EE (p = 0.003) and RLE (p = 0.018), 
while without the BC, RLEA was also higher than the rest of the treatments which included RE (p = 0.013), 
LEA (p = 0.027), and REA (p = 0.045). For Acquisition Total, RLEA was higher than LEA (p = 0.011), 
while without the BC, RLEA was also higher than RLE (p = 0.015) and RE (p = 0.026). For the Acquisition 
Mental Effort Rating, RLEA, REA, and RE were all lower than LE (p < 0.000, p = 0.001, and p = 0.009), 
while without the BC, RLEA was also lower than LEA (p = 0.006) and RLE (p = 0.009). The extremely low 
scores overall for Post-Test 2 among students make these last results less certain. Overall, these findings 
suggest that among the treatment groups, RLEA had the most significant positive impact on student 
performance during the acquisition phase. On the other hand, LE had the most negative impact. 


5. CONCLUSION 

The results of this study demonstrated students in a dual mode of instruction outperformed those in a single 
mode of instruction, but it is important to note the treatment groups that performed the best were engaged in 
tasks that required reading while the treatment group that performed the worst was engaged in tasks that 


281 



ISBN: 978-989-8533-43-2 ©2015 


required listening. These results can be explained by considering what Moussa-Inaty, Ayres, and Sweller 
(2012) called the transient nature of auditory material where additional processing may be required to 
remember previously heard information. The transient effect of listening can hinder learning since learners 
have no control over what they hear, which is not the case when learners read. But even though Moussa-Inaty 
et al. (2012), stressed that for foreign language learners, the simultaneous presentation of spoken and written 
text should be avoided, in the case of the current study, results showed that students benefited from learning 
how to solve mathematical word problems when written and auditory materials were simultaneously 
presented. Results also showed that when solving mathematical word problems, students performed better 
when two languages were presented simultaneously. 

In conclusion, this study supports the use of cognitive load -reducing strategies when learning how to 
solve mathematical word problems in a second language and this is particularly relevant in countries where 
English is the language of instruction but not the learners’ first language. When teaching how to solve 
mathematical word problems, the study can confirm that at least some form of reading should be involved 
and that auditory material should be avoided and not be presented alone as it may impose high cognitive 
load. The language in which mathematical word problems are presented in this study showed that it did 
impact student performance, but the unintended added time students in some treatments groups had, may 
have also been a contributing factor to enhanced performance. 
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ABSTRACT 

This work presents a new approach to the discovery, identification and connection of ontological elements within the 
domain of characterization in learning organizations. In particular, the study can be applied to contexts where 
organizations require planning, logic, balance, and cognition in knowledge creation scenarios, which is the case for the 
scenario we are proposing here as the primary one to consider in universities and academia. The model we are focusing 
on and proposing is grounded in the establishment of existing connections among people, processes, and technology, 
which form the compounds of the model being what we understand for the Experiential Learning Cycle (ELC) and the 
Imaginary Experiential Learning Cycle (IELC). Hence, a diagnosis method is proposed in the form of a ‘harmonograma’, 
which will serve as the basis to establish what we consider the Dynamic Balance Point (DBP) as a tool to be applied to 
identify the characteristics that mobilize cognition, and for which points need to be improved. Therefore, individuals (i.e. 
a person as a conceptualization) are concerned with personal knowledge, skills, experiences, and with the technology and 
processes that enable dealing and managing it. We certainly believe that the application of DBP as a self-diagnosis model 
could successfully contribute to this interaction of Technologies, Processes and People in the cognitive-behavioral 
perspective. 
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Ontology, experiential learning, dynamic balance point 


1. INTRODUCTION 

Davenport and Prusak (1998) have already stated that knowledge is an unrivaled asset, which is not 
consumed with use, but quite the opposite: through the use of existing knowledge it increases its value. This 
study presents a new approach to integrate relevant aspects of people, technologies and processes through 
Dynamic Balance Point (DBP) 1 

The theme will focus from the information theory to the consideration of the argument of the opponent. 
The argument of the opponents is that it is the functions of the brain that help in the process of perception 
between the inner and outer world. Together in this process is the interaction of gender archetypes. These 
archetypes of gender have spiritual and social power for civilizations that culture, and can be studied 
scientifically. These functions allow interaction with CAV (Experiential Learning Cycle), with the processes 
of perception that arise in interpersonal communication between the technologies and processes and people 
on the construction of cognitive behavior. 

I would divide into four types with clearer punctuation. In the structure of CAV, the triggers of cognitive 
behavior in human beings are the characteristics that mobilize entrepreneurial cognition. The characteristics 
that mobilize are divided into four main types: the awakening, which leads the skills that will help in 
planning, guiding the decision, which is a difficult issue because researchers are undecided on a solution, but 
the uncertainty principle is intrinsic to the event, the decision is a logical response to the needs of CAV the 
answer may not be applicable, the plan puts into practice free will through action: the structure, which 
consists of and symbolic logic, consistency with the practice of understanding in the process of kinesthetic 
action in behavior: running, whose character implies putting into practice the knowledge to improve the 


1 First introduced as PED from its original Portuguese translation (Ponto de Equilibrio Dinamico) 
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quality of life experienced in CAV. The theoretical basis of CAV was developed through experience gained 
in Solid State Physics (electromagnetic fields), the behavior of natural phenomena seen and understood 
through learning cycles in the TRANS-disciplinary process, applying inductive and deductive logic of 
knowledge in neuroscience. CAVI was theoretically established to encourage the study area of the thesis in 
the process of experimental psychology of perception he complexity of the behavior of the algorithm is not a 
time polynomial. An ontological relationship between CAVI and CAV was developed with the principles of 
the theory of quantum physics, hence arises a 50% probability of belonging to the phenomenon of CAVI, the 
other 50% belong to the phenomenon of CAV (uncertainty principle). However, cognitive-behavioral 
psychologies, physics, ontological phenomenology, were heuristic research and this research contributes to 
'trans-disciplinary events'. 

CAVI and CAV have contributed to the theoretical point of dynamic balance (DBP) among cultures 
(interpersonal relations), technology (the imaginary) and processes (for the syncretism of ideas) of being 
human. Concluding and first answer from CAV and CAVI, within the holistic vision of the cognitive study of 
alchemy, one can initially see the need to define the layout that flows and balance. CAV (the real world 
physical body) is experiential learning cycle, which begins in behavior, fulfilling the algorithm of the first 
cycle. 


2. CONCEPTUAL RELATIONS IN DYNAMIC BALANCE POINT (DBP) 

The Dynamic Balance Point (DBP) was coined by Oliveira (Oliveira, 2002) and it is a powerful way to 
manage, in an integrated manner, the interactive forces that exist when people, technology and process are 
involved. It is shown in figure 1 . 



Figure 1 . The DBP Model structure 

The DBP focus is identify mobilize characteristics related to people’s behavioral features like goals, 
persistency, planning and commitment (Oliveira, 2004). When working with people, we have the integration 
and balance between cognition and their ability through self-diagnosis. By working with technology, there is 
a DBP Model. This integration between people when they use technology is the cognition that occurs through 
the transformation of technology into desired results. When working with the process, there is DBP and this 
integration is the cognition that occurs through the methodology that was developed with flexible learning to 
acquire organizational knowledge. A final concept to consider in relation to knowledge management from the 
technological point of view would be the automated processing of knowledge, including through the use of 
an information technology system, in order to increase the value of collections of knowledge. The 
relationship between the People -Technologies, is a relation: [is - a], knowledge can be transformed by 
moving up the hierarchy of knowledge, a higher state in the same, that is, to what we might call "knowledge 
integrated" through a distributed system and easy access. As an example, we can transform the rules of data 
or cases using techniques of machine learning and data mining. The development of this rule is to give 
understanding of the philosophical questions that sometimes seem esoteric, but whose reasoning applied 
science in practice and often pass through pragmatic before going through the analyzers of our brain that are: 
Lexical - recognizes words; Syntactic-recognize the combination of words in the sentence; Semantic 
recognizes the meaning of words; Pragmatics the use of words. ’Human language uses sounds, symbols, 
gestures. Moreover, the signs of language are organized as a syntax-specific to each language or dialect. 'The 
language that compiles the human brain: is kaleidoscopic and for science can be seen as - Geometric 
Computing.' 
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3. DBP - MODEL ONTOLOGICAL RELATIONS AMONGST HUMAN 
RESOURCES IN PRACTICE 

3.1 Conceptual and Behavioral Relations in DBP 

Classical logic (first order) itself is a simple language with a limited number of basic symbols. The level of 
detail depends on the variety of predicates, which, strictly speaking, do not belong to logic. In this paper we 
will use the DBP Model and ontological methodology to model all the relevant entities that are considered 
and represented. Different varieties of predicates represent the DBP Model and the ontological commitments. 

For example: ’Person (P) is a (is-a) living, but every living being is a person 'Person is part of (part-of) 
of an organization’; ’Technology is part of (part - of) an organization', because technology is a (is-a) tool to 
achieve success. But not every tool is technology; ’Process is part of (part - of) an organization, it is a process 
(is-a) means to achieve success.’ But not all processes lead to success. ’Technology, when there is a failure, 
(is-a) problem to be solved by a HR (Human Resources) expert’; For DBP Model, staff management is 
unique because it depends on each HR (person - person or people - people). The framework of this model is 
Personal Planning and is different for each individual, but not separate from part of the universe, it is the set 
of personal values and behavioral, to which they relate. 

It then gives the relations between the elements described above as a method of confirmation of actions 
and plans chosen by the organization that is managed by HR. Cp equipment (a). It is a conceptual 
relationship of Person - Person. The HR is part of a team. Then, the behavioral relationships in the model 
DBP are satisfied. This does not mean they are alike, but belong to the same sets of successes/problems as ’a 
whole’, as part of an organization and/or project, sharing everything (people, technology and process). Let a = 
b, then b = a. The successes and/or problems are symmetrical, having the same degree of similarity. They are 
transitive: if a = b and b = c then a = c. C2: interpersonal (b). It is a conceptual relationship of person - person. 
HR are related in a civilized manner, it is a peer relationship, cooperation, integration and participation. C3: 
performance (c). In relationships, person - technology and/or person - process, HR relate, so that their 
motivation levels are very high, planned, structured and with a mastery of subject knowledge to solve. C4: 
Work (d). In the conceptual relationships. Person - Person, Person - Technology; Person - Process, 
Technology - Process, HR interact seamlessly with the DBP Model and ontological methodology. 

3.2 Ontological Relationships and Mathematical Properties of DBP: Dynamic 
Balance Point 

Ontological relationships and mathematical properties applied to Knowledge Manage are Human Resource 
practices in People - Processes, which are in balance in the model concept DBP work. In this concept 
described relations are satisfied with the respective properties as shown in table 1 . HR is considered an entity 
of the relationships that satisfies the properties of reflexivity, transitivity and similarity: People Management 
(process) is related to work (action) of HR. Examples of relationships that occur in the organizational context 
can be found below: - Equivalence between the concepts Person - Process: If the person (a) is member of a 
team (b), part-is the team (b) relates to the process(c); a R b a b R a a,b R c; - Topological between the 
concepts Person - Person, Person - Process, People - Technology (which fulfills the properties of reflexivity, 
symmetry and transitivity). These relations consider the infrastructure, the environment and machines; 
-Functional, between the concepts Person - Person, Person - Process and People - Technology. For example, 
a director (person) is the owner of an organization that relates to equipment experts (people) of quality 
software development (CMM-process) and great technology; - Purpose. For Human Resource to achieve its 
goals, it is necessary to know the purposes of the organization. With the globalization of the economy and the 
advancement of technology, organizations are under pressure in their production processes and negotiating 
new agents, less cost and greater efficiency, highlighting the cost -performance. 
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Table 1. Ontological relations and their properties. 


RELATIONSHIP/PROPERTIES 

REFLECTIVE 

MIRROR 

TRANSITIVE 

EQUIVALENCE 

X 

X 

X 

TAXONOMY 



X 

DEPENDENCE 

X 


X 

TOPOLOGICA 

X 

X 

X 

CAUSAL 




FUNCTIONAL 



X 

TEMPORAL 



X 

SIMILAR 

X 

X 


CONDITIONAL 



X 

PURPOSIVE 



X 

THINGS/OBJECT 




COMPONENT/OBJECT 



X 

MEMBER/COLLECTION 




LOT/EARTH 




LOCATION/AREA 



X 

ACTIVITES FEATURES 



X 

PHASE/PROCESS 



X 
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private space implicitly in Diagnostic Evaluation in Organizations (DEO). The indicators for diagnosis, have 
been grouped into four different sequence to develop work areas. DBP MODEL APPLICATION: Dynamic 
Balance Point among people, processes and technologies: 

HARMONOGRAMA DBP MODEL: The main objective of the methodology presented in this proposal 
is the determination of dynamic balance point (DBP) of Harmonograma namely the harmonic graph between 
organizational elements, whose scheme is shown in practice. The interrelationship between the vertices of 
Harmonograma has been the starting point for research that has developed through doctoral thesis at the 
Catholic University of Murcia-Spain. 


5. CONCLUSION 

Cognitive organizational education is similar to the DBP Model in quantum physics. Quantum physics works 
with imaginary, mathematical forms (CAVI), which represent physical possibilities in the real world (CAV). 
The DBP Model: Dynamic Balance Point between people, processes and technologies in cognitive education, 
with the possibilities that we can feel, associate, calculate, integrate and interact with processes in the 
transactions of life. We analyze the equilibrium of the body to what is defined by feelings, we classify CAV 
to the feelings and sensations CAVI, and consequently, what we call the sensation of CAVI to feelings that 
come from an energy transformation. We have cognitive sensation, which is the feeling of Organization in 
Education. The cognitive sense is the movement to find a balance point, which implies the existence of a 
mobilizing event. Education becomes an element that will evaluate feeling, which will shift more or less 
according to the process, and with people. 
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COGNITIVE CURATIONS OF COLLABORATIVE 

CURRICULA 
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ABSTRACT 

Assuming the role of learning curators, 22 graduate students (in-service teachers) addressed authentic problems 
(challenges) within their respective classrooms by selecting digital tools as part of implementation of interdisciplinary 
lesson plans. Students focused on formative assessment tools as a means to gather evidence to make improvements in the 
learning process. Floor plans of rooms in houses were used as metaphors by each group to contain the various digital 
tools that they selected. This paper includes descriptions of the final curations with samples of the digital assessments 
tools, as well as lessons learned for future applications. 


KEYWORDS 

Curation, assessment, digital, web2.0, online learning 


1. INTRODUCTION 

Content curation may be defined as the act of finding, grouping, organizing or sharing the best and most 
relevant content on a specific issue (Bhargava, 2011). Although we probably perform curation without 
deliberate thought, curation has only recently been applied to the learning (Gadot & Levin, 2012) and 
performance field (Fulton, Botticelli, & Bradley, 2011). Bhargava identifies five models for using content 
curation: 1. aggregation, 2. distillation, 3. elevation, 4. mashups, and 5. chronology. This paper focuses on 
graduate course assignments that extend beyond aggregating and filtering into value-added selections of 
digital tools to facilitate and assess learning (Rosenbaum, 2011; Mihailidis & Cohen, 2013; Callens, 2014; 
Green & Green, 2014; Literat, 2014; Greener & Wakefield, 2015; Ibrahim, Atif, Shuaib, & Sampson, 2015) 
with formative feedback to help people grow (Weisgerber, 2012; Niess & Gillow-Wiles, 2013). Topics in 
this paper include the background and definition of learning curation; a description of the audience and 
environment where the course was conducted; the course Assignment including the floor plan and rooms 
metaphor; web2.0 open sources, identification of digital tools that were selected for the container and 
instructional tools; screenshots of sample products; and results/lessons learned for future replication of the 
Assignment. 

1.1 Definition of Terms 

Curate - pull together, select organize, and look after items in a collection; select, organize, and present using 
expert knowledge; act as curator, organize and oversee; take charge of or organize 

Formative evaluation- range of formal and informal assessment procedures employed by instructors 
during the learning process to modify teaching and learning activities to improve student attainment 

1.2 Audience and Environment 

The paper focus involves 22 graduate students in a Master of Arts Instructional Technology program in the 
United States, with a course titled, Educational Flardware & Software. Participating students included K-12 
teachers (age 22-72) with three representatives from higher education, business, and the military. Class 
members were grouped into teams of 2-3 people who shared the same discipline. For student convenience, 
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class attendance options included face-to-face, online synchronous, online asynchronous (using Blackboard 
and Collaborate). The next section includes the Assignment given to the graduate students during a 10-week 
semester. 


2. ASSIGNMENT FOR CURATOR CREATION 

The term curator comes from the Latin term meaning take care. This term is widely used relating to a person 
who works in a gallery, museum, historical site, zoo, etc. and serves as the content specialist. The role of a 
curator in a museum is to arrange objects in a particular way that demonstrates a specific theme or tells a 
story. 

Recently, curators have become a popular term in the educational field as they play the same or similar 
role to a person in a museum. Since educators have access to exponential information it is hard for the learner 
to select what is important and what is not. As a result, the curator plays a vital part in the ecosystem of a 
learner by gathering and selecting the best examples to learn a particular concept and create the framework 
for the learner. 

The purpose of the educator is to assemble specific tasks to complete in a real, rigor, and relevant manner 
that is easy to navigate and engage all. In addition, infusing technology through this process enables 
educators to permeate global interaction through the use of social media. 

Throughout this course, you will serve as the Educational and Technology Curator. Your role is to work 
in a team (2 or more) who share the same discipline (science, math, language arts, social studies, etc.) or area 
of expertise. Each discipline will have its own floor (For example, floor one might be math, floor two is 
science, floor three is business, floor four is language arts, etc.) with customized rooms (Gaming Gallery, 
Fun Room, etc.) to meet the needs of the learner and the framework designed by the curator (Please see 
examples of possible rooms, yet feel free to create your own titles). 

2.1 Within each Floor, include a Minimum of the Following 

- Complete a lesson plan template integrating 9 events of instruction (See Gagne’ for further details) 

- Create a specific assessment using Bloom’s Taxonomy (authentic, MC/T/F, etc.) using a free web 2.0 
tool of your choice 

- Create an activity integrating social media that will increase collaboration 

- Create something (aka global wild card) on your floor that is original and not previously discussed 
(Each floor will be different) 

- Create a container or hub to house your information (iBook author, wiki, wix, weebly, livebinder, 
google site, eportfolio, edmodo, schoology etc.) for sharing with all of your classmates during our Synergistic 
Showtime. See Figures 1 and 2 for examples of room types to think about for your own floor plan. 

This slideshow may provide some background to get you started on what this is all about. 

http://www.slideshare.net/corinnew/reenvisioning-modern-pedagogy-educators-as-curators-11879841 



Figure 1. Room types 
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Every Room Must Have The Following: 

Creative Title: 

(Create a fun/engaging title) 

Problem: 

(Identify the learning gap in a specific area 
in the classroom) 

Purpose: 

(Briefly describe what the learner is going to do) 

Description: 

(Overview of room in 30 words or less) 

Linkage to Standards: 

(Common Core, NJCCCS, ISTE, ASTD, etc.) 

Infrastructure: 

(What does the learner need?) 

Operational Strategy: 

(How will learner accomplish task?) 

Evaluation tool: 

(Tool to measure learning outcome) 


Figure 2. What’s in each room? 



2.2 Examples Screen Shots of Curated Containers (Hubs) 


In this section, selected products are shown as screenshots within the container that the student used as a 
floor plan to house the various rooms to address his/her instructional problem (challenge). Figure 3 is a Time 
Management sample; Figure 4 is a Project Management sample; and Figure 5 is an Algebra sample. 


Types of Rooms 


M«t»coflniMon SoddMediTod I Cod Tod Asmiment 


Metacognition 

GPS: 

(Planning & Tracking 
Short & Long Term Goals) 

GPS 



Social 

Media 

Refrigerator Worthy: 
(Learning Styles & 

Refrigerator 

Worthy 



Effective Studying) 

Cool Tool 

Time Is 

Assessment 

Time Is Money: 

Money 


(Time Management) 


Lesson 

Lost & Found: 

Lost & 

Plans 

(Organization Skills) 

Found 


Figure 3. Time management sample rooms 
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We are all Project Managers. We schedule, track and manage our lives and our work, But, are we good at it? A project is a 
temporary, unique endeavor that requires attention to time, cost, and scope. Professionals lack the basic skills to 
successfully manage a project from inception to conclusion. 


Tokyo 

Disneyland 


'oiec 


]er 


"it's a small 
world" ® 

Global Collaboration 


World 

Showcase 


Disneyland ft 
USA 

Tim. Management 

Mr. Ray's 

Lesson Plan 


Keys to the 
Kingdom 



i*j 


Disneyland 

Paris 

Communication 

Disney 

World 

Problem Solving 

.1 

Imagineers 

Meet the Cast 


Test Track 

Assessment 



PODIO 



Social Madia 



Figure 4. Project management sample rooms 


Independent Learners In Search Of A Clue 

Heather Beattie 

Student Sample #2 



Container: http://beattieclue.weeblv.com 

Summary of project: 

http://www.voutube.com/watch?v=a3mGcPsT4bg&feature=voutu.be 


Figure 5. Switch to Independent Learners - Algebra 


2.3 Tool Selections 

As part of the Assignment, students selected a tool to serve as a container to house the rooms as well as free 
web2.0 tools for use in the instructional design to address the various challenges. The multitude of free 
assessment tools was an enlightening part of the project since the focus was to use formative assessments to 
help identify weak areas that could benefit from improvement. Sample tools they selected during the 
creation were as follows: scoopit, symbaloo, edmodo, quizlet, socrative, powtoon, emaze, pixton, edpuzzle, 
nearpod, pear deck, kahoot, quia, quill, zondle, quiz revolution, quiz bean, piktochart, creaza, twitter, 
wordpress, weebly, diigo, and pearltree. 
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3. RESULTS/LESSONS LEARNED 

As part of the formative evaluation of the project creations, these observations and questions emerged: 

- Many students ignored some of the directions such as word length, room components, creative titles, 
and thus lost points in the grading rubric. A question that may emerge is whether they curated the best or 
took short cuts to go for low hanging fruit — maybe a more in-depth rubric for tools selection could be used in 
the future. 

- Too many previous examples may have been available, and possibly confusing them (picking a handful 
of strong examples might be a superior method). 

- Check points - adding them with a Vi way freeze point in the semester may reduce procrastination. 

- Teams - working together may have contributed to a fragmented vs fully integrated, consistent approach 
for some products that evidenced format issues across containers (dividing work assignments in piecemeal 
without seeing the entire picture may have been a detriment in the long run). 

- Lack of clearly defined team roles and responsibilities, work plan... caused confusion, 
conflict/dysfunction. 

- Future plans — require creation of a team project plan in writing with agreed on charter, roles, and 
milestones to track the progress. 


4. SUMMARY AND CONCLUSIONS 

This paper included the background and definition of learning curation; a description of the audience and 
environment where the graduate course was conducted; the course Assignment including the floor plan and 
rooms metaphor; web2.0 open sources, identification of digital tools that were selected for the container and 
instructional tools; screenshots of sample products; and results/lessons learned for future replication of the 
Assignment. Many of the suggested revisions will involve more structured planning and implementation to 
keep the students on track so they may avoid becoming so engaged with the Assignment that they do not 
pace themselves for final housing of their curated products into their selected tool container. 
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HOW OLDER ADULTS USE VIRTUAL PERSONAL 
LEARNING NETWORKS TO SUPPORT INFORMAL, 
SELF-DIRECTED LIFELONG LEARNING GOALS: 
A RESEARCH PROGRAM DESCRIPTION 


Dirk Morrison 

University of Saskatchewan, Saskatoon, SK, Canada 


ABSTRACT 

This short paper will describe the details of a SSHRC/IDG-funded research program currently underway (2015-2016) 
that will investigate how older adults (65+) use Web 2.0 tools and Internet-based resources to establish and expand their 
virtual personal learning networks (PLNs) for the purposes of enriching their expertise and knowledge within the 
framework of their particular informal, self-directed learning goals. 


KEYWORDS 

Older adults; lifelong learning; virtual PLNs 


1. INTRODUCTION 

Global census data indicate a rapidly growing demographic trend: the percentage of older adults (age 60 and 
older) will rise in developed countries from 22% of the population in 2010 to 29% in 2030 (World Economic 
Forum, 2012). The Canadian context is parallel, with predictions indicating that 23.6% of the population will 
be over the age of 65 by 2030 compared to 15.3% in 2013 (CBC News, 2014; Kembhavi, 2012). 
Accompanying these demographic shifts are some sobering implications for society, including diminished 
quality of life for a large portion of the population and increased demands on human services. It follows that 
an important consideration for any society facing such challenges will be to identify and understand factors 
that favourably impact the mental and social well-being of older adults (Pike, 2011; Hare, 2014). While 
successful aging is multidimensional, the research is clear: maintenance of high cognitive function, sustained 
social engagement and the pursuit of meaningful activities are essential factors that contribute to our mental 
and social well-being as we age (Rowe & Kahn, 1997). Notably, informal, self-directed lifelong learning 
provides an effective conduit to achieve mental and social well-being; Merriam and Kee (2014) claim 
“learning in older adulthood not only reduces dependency on government-funded social services but actually 
enhances personal and community wellbeing” (p. 133). While sufficient evidence exists linking lifelong 
learning to increases in both social capital (Field, 2009), we have a very limited understanding about how 
older adults have transitioned to using Web 2.0 technologies to become more knowledgeable and socially 
engaged. This proposed research aims to explore how older adults use Web 2.0 tools to create and maintain 
personal learning networks (PLNs) to enhance their personal and community wellbeing. 


2. DESCRIPTION OF RESEARCH 

2.1 Research Context 

2.1.1 Older Adults’ Online: Informal Lifelong Learning 

While it is true our concept of lifelong learning for older adults needs to incorporate all types of learning, 
Merriam and Bierema (2014) claim: “most adult learning is through non-formal and informal means” (in 
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Merriam & Kee, 2014, p. 138). Non-formal learning includes structured but non-credentialed educational 
events or programs sponsored by community, civic, and voluntary organizations (e.g., Diabetes Society 
offering screening and nutrition education). Informal learning is that which occurs in one’s day-to-day living, 
and includes incidental learning but, importantly, also includes more systematic self-directed learning 
activities (e.g., a person designs his or her own learning project with concrete goals and outcomes) (Merriam 
& Kee, 2014). Older adults tend to meet a wide range of learning needs (e.g., health and wellness, leisure, 
personal interests, 21 st century computer and media literacy, etc.) via informal and self-directed methods. 
Evidence shows that, contrary to stereotypical perceptions of this phase of life, the majority of older adults 
are healthier, more economically independent, increasingly “computer-sawy” and that they actively pursue 
lifelong learning opportunities to enhance their lives (Kembhavi, 2012; Sadler & Krefft, 2008; Seals, et al, 
2008) or to “make meaning” of their lives (Fisher & Wolf, 2000, p. 483). Dench and Regan (2000) found that 
the most important reasons older adults gave for engaging in learning activities were intellectual : learners 
reported wanting to increase their knowledge, keep their brains active, enjoy the challenge of learning new 
things and learn about something they had always been interested in and for personal satisfaction (p. 1 ). An 
additional finding from the above research was that 80 percent reported a positive impact of learning on at 
least one of the following areas: their enjoyment of life, their self-confidence, how they felt about 
themselves, satisfaction with other areas of life and their ability to cope (p. 4). While such evidence is 
encouraging regarding the potential impact of lifelong learning activities on older adults’ mental, physical 
and social well-being, there is a paucity of research regarding older adult learners’ adoption and adaptation of 
Internet-based learning tools, techniques and resources that support informal, self-directed lifelong learning 
goals. Field (2000) warns us to not neglect such groups, risking exclusion, inequalities and erosion of human 
capital, and instead argues that we must embrace this widening participation in a learning society. This 
research proposes to investigate how older adults expand and enrich informal, self-directed lifelong learning 
goals via the creation, expansion and engagement of their personal learning networks (PLNs) using Web 2.0 
tools. 

2.1.2 Research Objectives 

The specific objectives of this research, followed by associated questions, are as follows: 

Objective 1: To investigate and describe older adults’ personal learning networks (PLNs): 

What Web 2.0 tools are included in the PLNs? 

How do they create their PLNs? 

Who they are connected to in their PLNs? 

What resources are used and exchanged in their PLNs? 

Objective 2: To investigate and describe older adults’ informal, self-directed learning processes: 

What are their motivations for learning? 

How do they set learning goals? 

How do they manage their learning (both content and process)? 

How do they communicate with others in the process of learning? 

What are the impacts of PLNs on their informal, self-directed learning? 

What are the impacts of these informal self-directed learning efforts on their personal and community 
well-being? 

2.1.3 Theoretical Approach 

This research draws upon a variety of theoretical constructs, conceptual frameworks and application contexts. 
Given the older adult demographic of this study, the use of adult learning and transformation theory 
(Baumgartner, 2001; Knowles, Holton, & Swanson, 2011; Meizirow, 2012), especially as it relates to 
informal, self-directed learning (Brookfield, 1985; Candy, 1991; Merriam, 2001; Marskick & Watkins, 2002) 
will be informed by a complementary background of theory and research specific to ageing and gerontology 
(Fishback, 1988/1999) to add power to analyses and interpretation of the data. In addition, because this 
research will investigate the use of Web 2.0 information and communication technologies, it will rely on 
current theory and best practices gleaned from Internet-based learning (Hill, Wiley, Nelson & Han. 2003) and 
social media applications in education (Hung & Yuen. 2010) to organize, synthesize and evaluate the 
findings. Finally, drawing on social capital theory as a component of the conceptual framework of this study, 
“the proposition that networks of relationships are a resource that can facilitate access to other resources of 
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value to individuals or groups for a specific purpose” (Balatti & Falk, p. 282) will ground examination of 
older adults learning together in online environments. 

2.1.4 Methodology 

The proposed program of research will utilize a mixed methods approach, with the collection and analysis of 
relevant qualitative and quantitative data from a representative sample of 384 Canadian older adult learners, 
65 years and older, who use Web 2.0-based personal learning networks to support their informal, 
self-directed learning projects. These participants will be invited to complete an online survey. 

To be included in the study, participants will: a) have had at least one online learning experience, b) have 
experience in using one or more Web 2.0 learning tools, such as PLNs, c) be willing to participate in the full 
research project, and d) be age 65 or older. Demographic data such as gender, time since retirement, marital 
status and level of educational attainment will also be gathered by the full online survey but will not 
influence selection for participation. 

Phase 1 of this research will address Objective 1: To explore and describe older adults’ personal learning 
networks (PLNs). The Principal Investigator (PI) and one qualified Master’s-level graduate student will work 
directly with the University of Saskatchewan’s Social Sciences Research Lab (SSRL) to design and construct 
an online survey, using the Qualtrics Online Survey Programming tool. Survey questions will be designed to 
address the questions outlined under Objective 1 listed above. The team will also consult with members of 
the Computers and Technology Lifelong Learning Program of the Saskatchewan Council on Aging (SCOA j, 
to solicit feedback regarding the survey instrument questions. 

Following the distribution and completion of the national online survey, analyses will include quantitative 
data and qualitative data. 

Phase 2 of the research will address Objective 2: To investigate and describe older adults’ informal, 
self-directed learning processes. This phase of the research will include 10 in-depth, online focus groups, 
consisting of five participants each, designed to examine participants’ qualitative descriptions of methods 
employed and tools used (e.g., Web 2.0) and perceptions of the relative value of these (e.g., PLNs) in support 
of their informal, self-directed learning goals. 

2.1.5 Data Analysis and Interpretation 

The rich variety of quantitative and qualitative data collected via the national survey and focus group 
interview transcripts across the two phases of research will be analyzed and interpreted with the goal of 
thoroughly and accurately describing older adults’ use of PLNs to achieve their informal and self-directed 
lifelong learning goals. Data analyses will be supported by a variety of tools and processes available to the 
researcher and graduate students such as participant interaction mapping and social network analysis 
methods. Focus group interview data will be organized, managed, and coded using the qualitative analysis 
software NVivo™ enabling a detailed, rich content analysis, and the discovery of complex relationships 
among and between these rich data sets. 


3. CONCLUSION 

This research explores older adults’ use of Web 2.0-enabled personal learning networks (PLNs) for the 
purposes of informal, self-directed learning and data from this new research will enable a rich analysis, both 
of the longer-term impact of PLNs on older adults, and of the relationship between individual use of PLNs 
and their general wellbeing. Non-profit and government-sponsored organizations whose mandate it is to 
serve the social and lifelong learning needs of older adults (e.g.. The Saskatchewan Council on Aging, 
Saskatoon. SK; Saskatchewan Senior Centre, Regina, SK etc.) could use such findings to better inform their 
programming initiatives. 
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ABSTRACT 

The research is conducted in a public institute of education and technology to boost graduation especially with the help of 
an e-learning environment adopted. The Directory of E-leaming Education from Federal Institute of Triangulo Mineiro 
coordinates all administrative and pedagogical aspects of 4000 students registered in 11 e-leaming courses, including 
federal government programs like e-Tec and Open University from Brazil (UAB - Universidade Aberta do Brasil). 
Although, e-leaming has helped to solve students’ mobility problems, there are remarkable problems in dropout rates of 
the virtual courses. The solution of the study aims to attenuate the dropout rates by adopting different learning 
methodologies, re-designing the learning environment and utilizing tools to support this new methodology. According to 
the hypothesis, the novel learning environment adopting a learner-centred approach, will give a voice for the students and 
give them an active role in their own learning process. Therefore, students’ needs, abilities, interests and learning styles 
are used to determine classroom activities, and consequently it will help to reduce the dropout rates. 


KEYWORDS 
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1. INTRODUCTION 

The problem of dropout rates in vocational and higher education levels has been a challenge for several 
educational institutions, teacher and students worldwide, and nowadays also at e-learning courses. The 
“digital natives” term is used to describe students of the 21 5,1 century, includes also a claim for new kinds of 
skills such as digital literacies. To help them, learning methodologies need to be changed too. There are 
studies reported in the literature where teacher make their approach more learner -centered by giving students 
more responsibility for their own learning. It gives voice for the students and. therefore, their needs, abilities, 
interests and learning styles determine classroom activities, and consequently it will help to reduce dropout 
rates. 

In learner-centered learning, students generally work in teams, defining their own roles and goals. 
Working with individual or group projects aiming to solve problems or, particular questions, make students 
more motivated, because they work with problems they have chosen. They learn with each other by looking 
for the best solutions for these problems. This type of learning is commonly called Project-Based learning. 

This paper presents the research about how to adopt these methodologies described above using tools to 
support it, applying in the e-learning environment (starting with a discipline that will serve as an example) at 
the Federal Institute of Triangulo Mineiro - IFTM. 

In an attempt to illustrate how the study will be utilized in this discipline, this paper is organized as 
follows. The literature review in the section 2 describes the factors that influence dropouts in e-learning 
courses and illustrates student-centered learning and project-based learning. Section 3 contextualizes the 
e-learning environment of IFTM and the statistical data about Brazilian scenario of dropout from e-learning 

courses. Section 4 describes the key changes to be applied at this discipline. Section 5 will discuss about 
the future works and the conclusion will finish this paper at Section 6. 
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2. LITERATURE REVIEW 

Starting this section, it is necessary to evaluate the factors that influence dropouts in Brazilian e-learning and 
to explore the skills necessary for the 21st century students. After these it will be possible to describe the 
approach that can deal with these challenges. 

2.1 Factors that Influence Dropout in Brazilian e-Learning 

Students’ dropouts is a complex phenomenon and it is common phenomenon in learning institutions around 
the world. It is a huge concern among directors, rector, researchers, parents and students. According to Silva 
Filho et.al. (2007) the dropout between undergraduate students is an international problem that achieve the 
learning system, and because this, it become study object around the world, including in 1 st world countries. 

There is a bigger concern about dropouts in e-learning, mainly because it is quite difficult for the students 
to deal with a possibility to study in a domestic, social or professional environment, and at same time let 
students to choose when they want to study. It occurs because there are concurrent stimulus (children, wife, 
television noise and neighborhood among others) at e-learning, and it depends more directly on students’ 
skills such as, organizational skills and concentration for studies. (Statistical annual report AbraEAD - 2007) 

According to AbraEAD (2007) and Lopes et al., the biggest reasons for dropouts in e-learning courses are 
related to time and financial conditions. Rossi (2008) shows in their researches that the dropout reason form 
the undergraduate students in courses at Open University of Brazil was time available and the disposal at the 
end of work day to conciliate the studies. 

According to Abbad et. Al. (2010 ) the factors linked to design of e-learning and blended learning may 
include following problems: the lack of compliance of personal expectations; the lack of information about 
the importance of the course; the low frequency of the use of web tools; dissatisfaction with the tutor 
performance; absenteeism tutors; the lack of assistance from tutor to student; the delay in sending feedback 
or supply a few information to students; physical separation between teacher and students; the duration and 
difficulty of the course; the mode of delivery of the course (difficulty accessing the website); the amount of 
written work required; the level of the course; the lack of face to face activity; the lack of support from the 
student teaching unit; the poor quality of teaching materials and the lack of interaction with colleagues. 

The fact is that more problems are related to the design of e-learning than in other characteristics 
presented above. Because of that, we intent to highlight some improvements to adopt e-learning courses at 
IFTM. 

2.2 Methodology Approaches: Student- Centered and Project-Based Learning 

According to Anderson (2010), adopting a student-centered learning in a classroom places a learner in a 
central position. The more learner-centered approach is applied by giving students more responsibility for 
their own learning. It gives a voice for students and. therefore, their needs, abilities, interests and learning 
styles determine classroom activities. Consequently it will help to reduce the dropout rates, because students 
are self-motivated. 

In student centered learning, students choose projects with which they like to work. Possibly, they define 
their learning goals, work actively to achieve these goals, and consequently they assess their learning. In 
learner-centered approach, students often work together in small groups, and they decide by themselves each 
member’s role in groups. 

When students are working actively and they have freedom to choose the subjects, they usually work with 
projects that focus in solving personal or interesting problems. The motivation is based on their background 
and experience. The problem solving of a particular subject or discipline is not limited and it may be related 
to other subjects and resulted in additional skills too. 

When project works are used as learning activities to seek for solutions to real-life problems, it’s 
commonly called project-based learning. According to Anderson (2010), information communication 
technology (ICT) rises an increasing interest in PBL, particularly the research opportunity provided by the 
internet and the array of multimedia tools for assembling and presenting the result of projects. 

The characteristics exposed before are convergent with the identified needs about factors that influence 
e-learning dropout in Brazil 
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3. IFTM E-LEARNING ENVIRONMENT AND HIGHLIGHTS 

The highlights to improve virtual learning environment of IFTM comes up with effective ways in facilitating 
desirable ICT skills that will be applicable to the students' education. There are four major objectives: (1) 
improve quality of assessments, (2) enable students to work in teams, (3) improve feedback about learning 
materials and (4) facilitate feedback for the students. These objectives were found and coincide with the 
emerging concept of Web 2.0 an so on. Within e-Learning development, the technologies that are being 
employed in various platforms are also changing. Many agree that a major characteristic of Web 2.0 is the 
enhanced social connection function of various web applications (Alexander, 2006; Anderson, 2007; 
0"Reilly, 2007). It is a more dynamic way of both accessing web based contents and connecting web users 
against the traditionally static and one way information web pages. Many also argue that Web 2.0 
applications provides simple to use and easy to maneuver social and networking tools. Therefore, it is more 
appropriate for creating community-driven and collaborative user experiences (Guzdial, Ludovice, Realff, 
Morley & Carroll, 2002; Chen, Cannon, Gabrio, Leifer, Toye & Bailey, 2005; Hampel, Selke & Vitt, 2005; 
Alexanda, 2006; Byron, 2006; Duffy & Bruns, 2006; Levi & Stone 2006; Chao, 2007; Parker & Chao, 2007). 

In professor Wilton de Paula Filho’s interview, he said that the main reasons to start modifications at 
IFTM learning environment was: (1) No pattern of virtual rooms in the subjects of the same course, (2) 
Feature incompatibility. E.g. A teacher offers chat and others not (3) Huge extension of sequential layouts 
(very large scroll bar) , (4) Constant complaint of students (due to the above mentioned points). The online 
tool to create the layout for virtual classrooms is free and it is available in 
www.personalizesuasalavirtual.com . Wilton is an e-learning teacher the developer of the e-learning tool at 
IFM. 

The standardization realized at the e-learning environment was very helpful, but there are other 
suggestions to improve quality of e-learning courses. 

The main existing problem is assessments and as an aim to solve it we decide to apply a formative 
assessment. Formative assessment is performed during the whole learning process, the opposite summative 
assessment shows the results at the end. There are several tools to be utilized in formative assessment. 
According to Vicki Davis, Socrative, Kahoot, Zaption, Backchannel Chat Tools and Plickers are favorite 
tools for formative assessment. 

The main idea is collaborative working with PBL, asking students to bring problems from their real life 
and community and it will motivate them. Another intention is to apply a SCRUM approach that is a project 
development approach. According to Rico and Sayani, the use of contemporary software development 
approaches such as agile methods is growing in widespread use throughout the world. Teams who struck an 
optimum balance of customer collaboration, use of agile methods, and technical ability, had better 
productivity. 

Feedback for students is usually given when they have finished activities, but we are planning to have a 
different approach. When the students have finished their mini lectures and watched the video tutorial about 
the subject, they will make the assessment. Of the psychological-sociological measurement methods that 
depend on human judgments, rating scale procedures exceed them all for popularity and use. Smith (2013). 
The purpose is that students can give feedback on all material that they used before and, ranking it to help the 
teacher to improve the material quality. In their conclusion. Smith (2013) currently suggests that the "best" 
scale for human voters should have 10 levels and consist entirely of nonnegative numbers ordered increasing 
from left to right. 

In other way, when the students are assessed using Kahoot or other formative tool, the ranking will serve 
to help the teacher give different feedback for the students. Those students whose assessment has low quality, 
will receive tips to be sent via Whatsapp. 


4. CONCLUSION AND FUTURE WORK 

Analyzing the context explained above it is easy to determine that the e-learning environmental alone will not 
offer the characteristics to motivate the students continuing their course to the end. 
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The old-school model of passively learning facts and reciting them out of context is no longer sufficient 
to prepare students to survive in today's world. Solving highly complex problems requires that students to 
have both fundamental skills (reading, writing, and math) and 21st century skills (teamwork, problem 
solving, research gathering, time management, information synthesizing, utilizing high tech tools). With this 
combination of skills, students become directors and managers of their learning process, guided and 
mentored by a skilled teacher. 

This short paper aims to explain how with some highlights on e-learning environment, we can improve 
the quality and to avoid dropout among undergraduate students. 

The next step is to apply the methodology and tools cited before in an undergraduate e-learning course 
and compare the result with the same discipline last semester. 
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ABSTRACT 

This paper describes E-fundi as a learning management system developed for the now more than 65 000 students of by 
the North-West University in South Africa. In this paper, it will be proposed that e-mentoring as a recent development of 
traditional mentoring, may be pursued by way of E-fundi to the enrichment and growth of students wherever they are. 
The advantages and disadvantages of e-mentoring will also be discussed. 

The paper will end by recommending ways the use of E-fundi for e-mentoring purposes may be implemented. 
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1. INTRODUCTION 

This paper argues for the use of a home-grown African developed learning management system called 
E-fundi to be used as an effective way to do e-mentoring. Applying E-fundi may be to the academic and 
personal advancement of the more that 65 000 students of the North-West University, 2th largest university 
in South Africa and where the majority of these students are off-campus. A short history and description of 
E-fundi will be undertaken, followed by a thorough discussion of e-mentoring and how that may be 
implemented towards students of the North-West University and other universities in Africa and globally. 
Reference will be made to similar recent research on e-mentoring which may be available as a further 
dynamic development of the age-old practice of mentoring. Advantages and disadvantages of e-mentoring 
will also be discussed. 

Finally recommendations how this may be introduced in other learning management systems starting 
from the available and existing E-fundi developed program and infrastructure. 

The contribution of this paper may lie in showing the possibilities of the extension and maximization of 
existing learning management systems to the holistic development of students in a university context, 
complementing the educational use thereof. 


2. BODY OF PAPER 

2.1 Background and Naming of E-Fundi 

Most universities have their own learning management systems which are named differently and may be 
developed locally or in co-operation with other institutions and/or organizations, often in a globalized 
context. The focus of this paper will be on the program being called E-fundi which is used by the North-West 
University in South Africa, where the author has been involved for nearly thirty years and the last few years 
have been exposed to the use of E-fundi. This involvement made one realize the possibilities of the system 
and how it may be applied even further and that something like implementing e-mentoring has been 
under-utilized totally. No specific research had as yet been done to show the possibilities of using E-fundi for 
the purposes of mentoring, but many faculty members use E-fundi as an indirect (and direct) way to mentor 
students with which they are in contact via E-fundi. The way it operates is where the lecturer pastes some 
information on E-fundi, immediately creates the opportunity to do mentoring via this communication system 
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where students are compelled to check into in a regular basis. Mentoring can then be done (in groups or with 
individual students), and the lecturer can decide which way the mentoring should go, while the students have 
access to it all along. Privacy with regard to mentoring of individual students is of course a matter of concern 
and caution should be taken about it. 

The development of E-fundi as the local learning management systems of the North-West University 
started in 2004 as per the following announcement (NWU, 2004 http://www.nwu.ac.za/content/new-online- 
learning-and-collaboration-tool-nwu): 

Education at the North-West University (NWU) will soon receive a new lease of life with the 
introduction of an online collaboration and learning tool, called e-Fundi. The tool, powered by 
Sakai (an online collaborative and learning project), is aimed at supporting teaching, collaboration, 
learning and research in higher education cycles. 

Joseph Hardin, director of Collaborative Technologies Laboratory at the University of Michigan 
and Board Chairperson of the Sakai foundation, was recently invited to the NWU to present a 
workshop on best ways to implement the tool. 

However, according to NWU project coordinator, Boeta Pretorius, the process of putting the tool in 
place is already underway at the university. "We intend to start with the implementation early next 
year, if all goes according to plan," he said. 

The name E-fundi is derived from the addition of the e- (for electronic) and the South African word fundi 
which means expert, authority, specialist, professional, master, pro (informal), ace (informal), genius, 
guru, pundit or buff (informal) (http://www.collinsdictionary.com/dictionary/english-thesaurus/fundi). 

2.2 Mentoring 

In introducing e-mentoring it is imperative to first explain what may be called traditional mentoring where 
mentoring actually started with. This has been done by people throughout the ages and the concept may have 
originated in Greek mythology some 3500 years back when Odysseus, the king of Ithaca asked Mentor, his 
friend, to take care of and teach his son, Telemachus when he went off fighting the Trojan war (Lotter, 
2007:1). 

Traditional mentoring always had the following components: someone who is the “senior, older partner” 
taking the lead and training or guiding the younger, less experienced person who follows the mentor’s lead. 
The mentor is the more experienced person and the one being mentored, is called the “mentee”. As Tucker 
(2007:iii, vi) aptly states: 

“Mentoring is a supportive, learning relationship between an individual -the mentor-who shares his or 
her knowledge, experience and insights with another less experienced person -the learning 
associate-who is willing and ready to benefit from his exchange 

According to Lotter (2010:6) some different forms of mentoring can be used in circumstances: whatever 
the situation requires be it informal , formal or in other spheres of life, like in a family context. Mentoring 
may be voluntary or it may be within a very rigid, tough structured business environment where it is enforced 
by “from above”. 

2.3 E-Mentoring 

As with many developments in the “digital age”, e-mentoring also grew from traditional mentoring and has a 
life of its own, separate from the origin of traditional mentoring. 

Law, Ireland & Hussain (2007:16) explained: 

“Another area where the Internet has developed our thinking is in matching pairs of people for 
collaboration. . ..One of the biggest challenges is in matching learners at a distance and across cultures. The 
interpersonal relationship between mentees and mentors in which a safe place exists for the mentee to grow is 
important.” When writing about E-Mentoring as virtual replacement of traditional mentoring, Lotter 2008:8 
also states: E-mentoring started to gain popularity in the nineties when teachers realized that e-mails may be 
an easier and better way to connect with students around inter alia curricular projects, research and preparing 
students for business. 
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According to Williams & Kim (201 1:80) e-mentoring may either be a relationship, a method or perhaps a 
process and/or an organizational approached which may be practiced in relationships. 

A preliminary definition of Lotter (2008:7) on e-mentoring reads as follows: E-mentoring is some kind of a 
relationship that exists between a mentor and someone who is less skilled or experienced, called a mentee, 
where electronic communications are used, and this interaction is focused on developing the mentee 
(see also the more comprehensive definition of Single and Muller, 2001). 

If one only checked the amount of postings using the word e-mentoring on the internet, one realizes how 
vast this “new” field is and constantly growing as the technology advances. 

Where e-mentoring has been used at certain schools as a helpful tool, excellent results followed as is 
reported in the following quote where Heynes already back in 2007 describes e-mentoring as follows: 
E-Mentoring is a viable and useful additional tool to add to the armoury of all schools who seek to 
raise achievement. There are significant academic and social benefits of involving older students, 
PGCE students and staff in e-Mentoring. The costs of e-Mentoring are not prohibitive and the 
technology is robust and user friendly. 

As with many developments in the “techno” - world, all new inventions are not necessary a blessing (c/ 
the paper by Lotter [2006] titled Dehumanization by cyberfication), and one realizes that with e-mentoring 
there are advantages and disadvantages as will be discussed below. 

2.4 Advantages and Disadvantages of E-Mentoring 

The following advantages and disadvantages may be applicable in the use of e-mentoring: 


Table 1 


Advantages 

Disadvantages 

• Mentor and mentee may be connected most 

• One-sided withdrawal by one of the parties 

of the time 

(especially the mentor) may let the other 

• Accessibility 

feel rejected 

• Immediacy 

• Not personal 

• Useful tool without taking too much time 

• No connection possible without some kind 

• Everywhere available, even globally 

of electronic apparatus 

• Availability since via cell phone one can be 

• Co-dependency can develop, where both 

reached anywhere 

parties are in permanent contact with each 

• Different forms of communication: e-mail. 

other and cannot cope without this contact 

text messages (SMS) or MXit (South 

• Boundaries may be crossed 

African), WhatsApp and skype 

• Invasion of privacy on both sides is a real 

• Psychological and other support a click 

danger 

away 



3. RECOMMENDATIONS 

Definite boundaries and mutual agreement should be set by the mentor and the mentee in the 
e-mentoring process keeping the following in mind. Recommendations regarding these issues may be the 
following: 
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It should be determined that the learning management system is geared towards the dealing 
of e-mentoring traffic 

To be successful it will be an advantage if the institution supports the system of e- 
mentoring 

There should be a mutual agreement about the contents of the e-mentoring being followed 
Boundaries should be set by both parties 

An imperative of respect for privacy and hours should be maintained 

The possibility of a personal meeting (if viable) should always be open to get feed-back on 

the success of this specific way of mentoring 

Any problems or outstanding issues which may exist, should be ironed out directly and 
immediately 

It is crucial to agree on the duration of e-mentoring regarding the years, project, equipping 
and other relevant matters 

There should be the freedom to move out of an e-mentoring situation and relationship at 
any time one of the parties would choose to do so 


4. CONCLUSION 

In this paper it was shown how E-fundi as a locally developed learning management system (in co-operation 
with Sakai) was created and now serves more than 65 000 students of the North-West University in South 
Africa. It was explained where traditional mentoring came from and how e-mentoring developed within the 
parameters of the digital enhancement. This rapid development also opens the possibility to use E-fundi for 
the enrichment and growth of students within a mentoring program anywhere they may be reached in the 
cyber world. The advantages and disadvantages of e-mentoring had also been discussed. 

The paper ends by recommending ways the use of E-fundi for e-mentoring purposes may be 
implemented. 
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ABSTRACT 

Recently, Considerable advancement has been seen for educating the people with autism. Research has shown that people 
with autism reveals positive behavior while interacting with innovative information technologies as compared to 
therapies [5], This review focuses on the possible use of ICT in the education and development of the people with autism 
and reducing their difficulties in communication [18]. A variety of different ICT applications for the treatment of autism 
including the use of interactive virtual environments, devices, computers, serious games and avatars will be discussed in 
this paper. ICT is a rapidly developing area of research to be used as a therapy for autism. Despite the fact that ICT has 
exciting positive results, most of the ICT proposals or implementations have limited use, performance and capabilities in 
actual conditions [6, 18]. Interactive devices, including robotics are being developed and used for the education and 
development of the people with autism. These devices teach and practice a skill, promote behaviors and provide feedback 
in specific environments. Development of interactive devices including robots having humanlike abilities and joint 
attention are the issues that are to be focused in the future research for assisting the people with autism [14], 


KEYWORDS 

Technology, Autism Therapy, loint Attention, Learning Disabilities. 


1. INTRODUCTION 

It has been proved that in the development of a child, social-emotional interactions play an important part [5, 
18]. Communication is the key to develop the ability to identify, translate and create behaviors as well as it is 
the basis of social cognition. In social cognition, a child interprets verbal and non-verbal communication 
nodes like speech, gestures, facial expressions etc. [10]. It also helps a child to produce such nodes as social 
and emotional movements, expressions and gestures as well as synchronizing their behaviors with others and 
the ability of giving everyone a credit according to their state of mind [2], 

There are two areas where disorders related to Autism are defined: (1) Permanent problem in social 
interaction and (2) Repetitive behavior, activities or interests [9, 10]. The main problem in autism is the 
difficulty in communication and interaction with others due to the lack of understanding or interpreting the 
expressions, gestures or movements [3]. Especially in children with autism it is observed that they prefer to 
play alone with their repetitive activities instead of playing with others. So it is big problem for them to keep 
interacting with human and the environment [6, 9]. Although it is a type of disorder in which positive 
development is poor but still considerable advancement has been observed in the development of such people 
with the help of various approaches [18]. 


2. ICT Applications 
2.1 Use of Computers 

The focus of ICT applications is to develop an interactive relationship between one user and one computer as 
well as to improve behavioral disorders related to autism. As computers provide predictability and 
consistency, so they are helping to motivate the children with autism as compared to humans. Computer is 
reliable for not sending social messages which create confusion [5], From the research on students with 
autism, we concluded the increase in the following: (a) Focused and overall attention (b) Fine and 
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generalized skills and behaviors and decrease in the following (a) self-stimulatory behaviors (b) nervousness 

[3]. 

Assistive technology for autism can be used for improvement in daily activities. Hetzroni and Tannos 
conducted a research on five children and developed a program based on daily activities keeping the 
communication in main focus. This program showed improvement in learning as well as communicating that 
lesson in the classroom. ARC (Autism Research Centre) launched an educational emotion software, named 
Transporters. The Transporter is being used commercially for autism. It is a vehicle based on eight characters 
which attracts the children while moving due to its mechanical nature. Transporter containing faces of actors 
with emotions was developed to evaluate the ease in learning as compared to real world. Transporters 
resulted that this DVD is an effective way to teach the children to recognize emotion and can be generalized 
to new situations with new faces on the vehicles. Other who were not taught by Transporter resulted in less 
improvement [18]. 

2.2 Virtual Environments 

Over the last decade, many software related to virtual environments have been produced for autism. VEs 
have the predictable interaction which minimizes the anxiety. The safe and close to nature scenarios of VEs 
as well as the animation and voice helps the children to take interest which minimizes the learning disabilities 
in a better way [11], 

In past experiments. Autistic children have proved the successful learning of skills with the help of VE 
and also enjoyed the technology. Children, especially, are more interested in VE as virtual peers are 
animated characters with language enabled, computer generated, responding and life-long instead of other 
toys. Story telling scenario is the most researched area of this technology which gives company to a child 
[15]. In contribution to this research, a virtual cafe in which participants have to order a drink, sit and pay 
was programmed. This was a reality program to understand the social behaviors and examined on the 
children of teen-age. Different activities were taught and practiced including finding seat in both empty and 
crowded cafe and communicating with others in the cafe. It was concluded that more complex programs may 
provide children with more realistic interaction with social knowledge [17]. 

Another program in hie a virtual supermarket was created to make children think and play with touching 
the screen and interacting with objects. This increased the imagination and symbolic thinking of children and 
the outcome was more positive in sense of understanding and interacting. The multidimensional interaction 
with touch screen improved the participation of both learners and instructors with natural interaction between 
computer and child, instructor and child and computer and instructor as well [15]. 

A multi-user, multi-touch display. Diamond Touch Technology, can be used to engage the children in 
imaginative scenario where all the actions have real consequences in VE. This technology was further used 
for the improvement in positive social behaviors including eye contact in terms of both quality and quantity. 
This concluded that touch screens can be used to promote creativity and in future this can bring great 
advancements in learning of autistic children [17]. Although it is a reality that VE does not provide real touch 
of texture and feelings, sensing but still it is a promising improvement for development of disorders as this 
trains and motivates the child for the real-time interactions [15]. 

2.3 Interactive Environments 

Recent researches have bring advances in this field of collaborative interactive environments through the 
control and monitoring of the behavior of the autistic child. This brings the skill of collaboration as well as a 
safe and enjoyable experience for the children because of no judgment issues in such interactions [18]. This 
is a notable thing and also proved that use of computers provide attractive environment for the purpose of 
education to an autistic child whereas real-time communication can lead them to problems. During the 
process of educating, different states of the behaviors of the child are measured with time in case of 
accomplishment of any goal and can lead to the factors involved in the progress of the child [2]. 

It becomes more complex and difficult task to educate a child with ASD in real-world interactive 
environments when the trainer has to think rapidly [1]. Many environments have been developed using 
pictures and sketches to entertain the child so far with the help of education platforms. Pictures and sketches 
are used to make the child distinguish the size, shape and color of the things as well as with the help of sound 
and voices. This type of environment also minimizes the monitoring of teacher due to their verbal and visual 
guidance [15]. 
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2.4 Special Input Devices 

Recent developments of ICT allow more attractive form of input after the research that people with ASD 
enjoy interacting with computers. That is why now most of the technologies use touch screen instead of 
mouse or joystick for input or feedback. In recent research, a multi-user touch was introduced that allows two 
or four user and detect their touch with the help of an antenna activated by all users one by one [17]. 3D 
environments, remote controlled environment and interfaces with big colored buttons increase the process of 
development in autistic people. With all such technologies, external devices are used to monitor and evaluate 
the internal and external state of the user. These external devices can be a wearable measurement device, a 
camera or a data glove. Video projectors are also an example of attractive virtual worlds that are being used 
by scientists for the educational purpose to be depicted on a wall [12, 13]. 

In history, TEACCH was the first program for educating the autistic children which have the principles 
like behavior changing and development of skill according to the personal need of the person. By recording 
the education process, these recordings can be further used to educate the other cases of ASD as well even by 
modifying the process according to the special needs of an individual. Apple, ipads, ipods and similar devices 
have been used to educate the individuals with ASD [18]. 

Studies were made on the following domains: (1) employment (2) academic and (3) leisure aimed to (a) 
deliver instructions and (b) teach to operate the devices. Half of the studies gave positive feedback as well as 
increase in their level of understanding and recognizing the objects. These studied proved that video 
modeling on ipads can be an effective technique to teach the students with ASD [16]. Personal Computers, 
mobile phones and MP3 Players are evaluated for the purpose of communication meditation of ASD. 
Research related to ipads for communication purpose and their comparison with communication system 
using picture cards was done by different authors. Similarly, the story telling video clips were used to 
increase the behavioral skills in the ASD individuals [17]. 

2.5 Serious Games 

Serious games play an important role for the development of ASD as the instructor. Additionally, serious 
games playing improves the social skills and decision making of the participants [8]. Autistic people are 
found spending most of their time in games or cartoons and according to their parents, they are aware of 
cartoons more than people [15]. A game, named Jestimule was developed with heterogeneity to facilitate the 
children with ASD. Moreover, ICT was also used in a game to teach all kind of students the ability to 
recognize the emotions and all kind of situations effectively. With the help of this game, the participants 
improved their ability of facial and emotion recognition in different situations. Such kinds of results helped to 
improve these technologies for the use in future to train the autistic children [18, 12]. 

2.6 Telerehabiliation 

The method of using telecommunication, operation and computing technologies for the purpose of educating, 
training or development of the autistic children by serving clinics, clients or systems for assisting them. 
Comparing the telerehabiliation with traditional approach resulted that telerehabiliation provides more 
effective ways in creating the supportive environment. In recent years, the biggest advancement have been 
seen in the field of telecommunication which is a backbone for the telerehabiliation [10]. 

In a comprehensive review report on the telerehabiliation and its models defined many health services 
that can be provided across distance which was named as telehealth. Three subcategories of telehealth are: 
(1) Telehealthcare (2) Telemedicine and (3) e-health/education [17]. The main focus was on the improved 
telerehabiliation with minimal cost, time and fastest access to the experts. Through this, various technologies 
involving geographic and economic problems were used for cost-effective health providence in home. This 
research also showed that telerehabiliation at home is as effective as the service providence in hospitals. 
These services can include cognitive rehabilitation with the help of internet or movement therapy with the 
help of using a computer/sensors for guiding through exercises [5, 6], 

Most of the interesting or beneficial part of telerehabiliation is seen for the autistic people. Research is 
being done on this technology to provide assistance for the ASD at home by interacting with the therapists. 
Some barriers are being faced including the lack of specialists with long list of awaiting persons as well as 
the distance barrier [6]. Research is continue on telecommunication to provide health assistance to families at 
home by removing the distance barriers [1, 5]. 
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By study, we have examined the telemedicine technology used to provide parental guide and teaching 
modality to support their children in social, communicative or play development. Guide was provided 
through a video conferencing program or coaching the parents through internet with the ability to see, hear 
and communicate as well with the specialists [5]. 


3. CONCLUSION 

ICT research has explored many ways for the education, development and treatment of autistic people 
including the followings: 

(1) Assistive technologies and their impact on daily life in cognitive impairment (2) improvement of 
social cognition through cognitive rehabilitation/meditation and (3) Cognitive computing to help in acquiring 
social and academic skills [1, 19]. Much of the things are still needed to be explored to have success from the 
perspective of practical treatment. Most of the ICT applications are limited in performance because of their 
unnatural interaction and use. Recent research reveals that use of computers. Avatars, serious games, 
telecommunication interactive and virtual environments generate a high degree impact of motivation and 
teach a person with ASD or learning disabilities specifically a person who is not willing to be social [6, 7]. 
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ABSTRACT 

Digital technologies are being increasingly used in wider society including in educational settings. There are many 
examples that illustrate how universities embed technology enhanced learning within their educational provision. 
Elowever there has been less research and evaluation of how these and other readily available technology based resources 
are being used in practice by students. This study aims to provide an in depth understanding of how a specific subset of 
higher education students, international STEM undergraduates in a large UK university are using digital technologies to 
support their studies. The results of a survey with a small group of students are reported. Initial findings show that 
students use technology in both formal and informal settings drawing on university resources but also other widely 
available resources. They are also routinely using more than one device to support their studies. Although they often use 
technology to support collaborative learning, more formal systems such as university provided discussion forums are not 
widely used. The results suggest that higher education should evaluate its use of digital technology and digital learning 
environments to determine if they are fully supporting their learners’ needs and are effective. Furthermore higher 
education should consider how students are already using other widely available digital technology resources to support 
their learning and what they can leam from this to inform their own programme design and delivery. 
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1. INTRODUCTION 

Today digital technologies often play a key role in facilitating and supporting students’ learning. Digital 
technologies encompass the range of electronic tools, systems, devices and resources that generate, store or 
process data and information including social media, online games and applications, multimedia, productivity 
applications, cloud computing, interoperable systems and mobile devices (DET, 2014). ‘Technology 
enhanced learning’ or ‘eLearning’ are terms often used to refer to a type of learning that is facilitated and 
supported by digital technologies. The use of digital technologies can not only enhance traditional ways of 
teaching but also can expose learners to new and different ways of learning. They can facilitate a shared 
learning environment by enabling learners to collaborate in establishing communities of learners that go 
beyond the classroom; support the formation of learning environments and resources that cater for different 
learning styles and approaches; and, ultimately provide students with different learning experiences. They 
can also provide an opportunity to better understand students and the way they learn. 

In recent years UK universities have set out to attract significant numbers of international students. In 
2012-2013, around one in eight of all students enrolled in higher education in the UK were from outside the 
EU (UKUniversities, 2014). They now make up a significant number of postgraduate student numbers: one 
in four with a total of 91,485 students in 2012-2013 (UKUniversities, 2014). There are also significant 
numbers at undergraduate level, 69, 640 in 2012-2013 (UKUniversities, 2014) but there has been less focus 
on how to support this group of students. Anecdotally it has been observed that these students often use and 
share digital resources such as youtube, facebook and other social media to support their studies. They are 
also sometimes unaccustomed to the UK’s higher education approaches to learning and teaching and can find 
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them strange and potentially challenging to engage with. This study sets out to provide a more in depth 
understanding of how these students currently use digital technologies to support their learning and what can 
be learnt from this in order to better support these students in their studies. It focuses on a set of international 
undergraduate students at a large UK university studying a range of STEM (science, technology, engineering 
and mathematics) disciplines. This paper reports on the initial findings from this study. 



Figure 1. Framework on Digital Technologies and Informal and Formal Learning 


2. RESEARCH STUDY 


2.1 Research Approach and Framework 

The study investigates how students learn in both formal and informal settings. Formal learning sometimes 
also called university learning refers to learning that takes place in formal settings such as university or 
tertiary institutions and is highly structured in its curriculum learning activities and assessment and usually 
leads to a qualification (Lai et al., 2013). It is more difficult to define informal learning, which some regard 
as all learning outside the classroom, while others regard it as “a self-directed, intentional interest ( rather 
than curriculum-based), non-assessment-driven and non-qualification-oriented endeavour ’ (Lai et al, 2013). 
There is a growing recognition that a semiotic relationship exists between formal and informal learning and 
that a student’s informal learning can be triggered by their work done in the formal education setting (Gurung 
& Rutledge, 2014; Laurillard, 2009; O’Mahony, 2010). For the purposes of this study, informal learning 
refers to learning that happens outside the formal learning environment and support mechanisms provided by 
the university to support students on their programme. In informal learning situations, whilst learners use the 
learning strategies gained from formal settings, they also use strategies that are not normally used in 
university. 
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The concept of seamless learning is useful as it encompasses the transitions between different learning 
situations and contexts (Looi et al., 2010). Drawing on previous studies in this area and the integration 
between formal and informal learning, (Gurung & Rutledge, 2014; Lai et al., 2013; Looi et al;, 2010; 
Margaryan et al., 2011; Osborne & Dillon, 2007; Sefton-Green, 2004), the authors have drawn up a 
framework which outlines the main critical success factors for technology enhanced learning and the key 
activities that underpin formal and informal learning (see Figure 1). This framework has been used to design 
a questionnaire, the first stage in this mixed methods study. It was distributed among a small cohort (n=l 1 ) of 
international undergraduate students studying a range of programmes across the STEM disciplines. A 
research survey approach is often used for studying behaviour and thus is appropriate for this study where the 
authors wanted to explore the following questions: “ what digital technologies do international 
undergraduate students use to support their learning?” and “how do they use these digital technologies to 
support their learning? ” 

2.2 Results and Key Findings 

The students were asked which digital tools they use both in a formal classroom setting and outside the 
university. The results indicate that they use the virtual learning environment, internet, email and standard 
office suite regularly in both settings. They make very little use of discussion boards and 
blogging/microblogging (e.g. twitter). Video sharing such as youtube, wikis, ebooks and document sharing 
are all popular and used more frequently in an informal setting. Table 1 illustrates the use of different types 
of devices by the students and clearly shows how they routinely use more than one device to support their 
learning. Students were also asked how many hours they typically use technology on and off campus. The 
results from this question are shown in Table 2 which indicate they spend more time off campus using 
technology than on campus. The students were asked how they usually work with other students on their 
course and/or share ideas with them. Although face to face is still popular, email, messaging (e.g. SMS) and 
social media (e.g. facebook) are all popular too. Students were also asked to indicate the main purpose of 
their use technology to support their learning. Their responses, shown in Table 3 indicate that interaction with 
others is rated highly, both in terms of sharing practice/resources and in seeking help to queries. When asked 
what issues they had with using technology, the general response was low indicating they had very little 
issues. However they do seem to make use of helpdesk support and online help, showing they seem to be 
able to seek out help themselves to any issues they experience. 


Table 1 . Frequency of Use of Digital Devices by Students (out of a total of 1 1 ) 


Type of Digital Device 

Use Daily 

Use Weekly 

Use Monthly 

Never Use 

Desktop computer 

7 

4 

0 

0 

Laptop 

10 

1 

0 

0 

Mobile phone 

11 

0 

0 

0 

Tablet 

2 

1 

4 

4 


Table 2. Frequency of Use of Digital Technology by Students (out of a total of 1 1) 


Hours of Use 

1-3 hours 

4-6 hours 

7-9 hours 

10+ hours 

On Campus 

6 

1 

1 

3 

Off Campus 

2 

3 

4 

2 


Table 3. Main Purpose for Using Digital Technologies 


Purpose of Using Digital Technology 

Response (out of 11) 

Communicate with other students 

10 

Share resources among students 

9 

Evaluate the work of others 

2 

To support formal assessment 

3 

To ask questions 

10 

To engage in discussion 

8 
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3. CONCLUSION 

This paper reports the findings from an initial survey of undergraduate international students and their use of 
digital technologies in both formal and informal settings. The authors constructed a framework of success 
factors and key activities from previous work in this area and this proved useful for designing the survey for 
the students. The results from this survey indicate that these set of students are routinely using a range of 
digital tools and devices to support their learning and that they experience little difficulties with their use of 
technology. Although they use face to face communication for working with others, digital technologies such 
as social media and email are also frequently used and they value the opportunity to work with others. A 
limitation of this work is the small number of students surveyed and further work is underway to conduct a 
wider survey combined with a more in depth qualitative study to seek out a deeper understanding of the 
approaches and reasons behind this set of students’ use of digital technologies. The results from this and the 
follow on study will be informative for higher education institutions in how they use digital tools and 
technologies to support students with a similar background. 
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ABSTRACT 

This paper proposes a novel e-Learning system for learning electronic circuit making and programming a microcontroller 
to control a robot. The proposed e-Learning system comprises a virtual-circuit-making function for the construction of 
circuits with a versatile, Arduino microcontroller and an educational system that can simulate behaviors of robots based 
on programs written by individual experimenters. The proposed system is useful and effective for comprehensive studies 
on electronic circuits with sensors and for programming robot control. The usefulness and effectiveness of the proposed 
system were evaluated by five university students. Results showing positive responses, which indicate the usefulness of 
the proposed system, were obtained from all students. 


KEYWORDS 

e-Learning system, virtual circuit making, microcontroller, construction and programming of a robot 


1. INTRODUCTION 

Teaching and learning both hardware and software are necessary in the field of technology education. 
Learning to construct an electronic circuit with a microcontroller (microprocessor) and to program machine 
controls (e.g., robotics) is effective in the study of technologies and increases student motivation to learn 
engineering. Figure 1 shows techniques needed to construct a robot and to control its motion. Students in the 
field of technology have to efficiently learn all techniques (1-5) shown in Figure 1. The LEGO Mindstorms 
system (http://www.legomindstorms.com/) is the most widely used for many educational purposes 
because it is relatively inexpensive, reusable and programmable (Behrens, 2010; Gomez-de-Gabriel, 
2011; Kim, 2011; Takemura, 2013). The LEGO Mindstorms system enables users to easily construct various 
types of robots using a minicomputer module (RCX, NXT, or EV3) and plastic blocks, and users can study 
the programming that controls the robots’ motion. Therefore, the LEGO Mindstorms system has been used to 
teach engineering technologies to students from primary to graduate levels. Flowever, conventional 
LEGO-based systems are not sufficient to use for technical education because of the disadvantages as 
follows: 

•The minicomputers of LEGO Mindstorms are black boxes. Therefore, students cannot learn 
electronic circuits used in the robots. 

•Each minicomputer (RCX, NXT, and EV3) needs proprietary software for programming, and the 
programming environment is not compatible with the different types of minicomputers. 

Moreover, conventional education systems on robotics are not sufficient for comprehensive studies 
involving technological elements (1-5) shown in Figure 1; and the conventional systems are not applicable to 
e-Learning. To overcome these disadvantages, this paper proposes a novel e-Learning system for 
comprehensive studies of robotics, including all technological elements shown in Figure 1 . Specifically, the 
proposed system comprises the technical functions as follows: 

(a) A virtual-circuit-making (VCM) function for the virtual construction of a circuit with 
sensors and a generally used Arduino microcontroller (http://www.arduino.cc/). 

(b) An e-Learning environment to study Arduino programming to control a line-tracer robot. 
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(c) A learning tool that can simulate robots’ motion based on the program written using 
function (b). 

(d) Compatibility with various types of computers and operating systems. 

A microcontroller is a small programmable computer board with an embedded integrated circuit 
(microprocessor), and using a microcontroller is an effective method for students to learn machine 
manufacturing and programming for machine control. Arduino is a widely used microcontroller, and 
open-source programming software for Arduino is generally available. The proposed system enables users to 
study the construction of practical circuits and programming using Arduino without exclusive software. The 
system was evaluated by undergraduate students in a class at the Tokyo University of Agriculture and 
Technology (TUAT). 


■ Hardware 


■ (1 ) Sensors 

• Function as a sense organ 


4 


(2) Electronic Circuit 


r 


Software 


(4) Programming 


4 

(3) Micro-controller - 
• Function as intelligence 



f (5) Machine Control A 
• Robot motion 


Figure 1. Schematic of technological elements necessary to teach and learn the construction of robots and their motion 

control. 


2. METHODOLOGY 

To cover all necessary technological elements shown in Figure 1, the proposed e-Learning system comprises 
the VCM function, circuit recognition function, and translation and simulation functions of the Arduino 
program. Technological features of these functions are described in Sections 2.1— 2.3. To evaluate the 
proposed e-Learning system, experiments on a line-tracer robot were conducted by university students 
(as described in Section 2.4). The line-tracer robot can detect the edge of a thick black line on a white 
floor and move along the black line. 

2.1 Virtual- Circuit-Making Function 

The VCM function is useful for users because they do not need to work with physical circuit 
components to learn designing and construction of electronic circuits. The VCM, proposed in a 
previous study (Takemura, 2013), enables individual learners to use graphics editors, which are installed on a 
learner’s computer. Therefore, this system does not require the use of proprietary graphics software. To 
construct a virtual circuit of a line tracer, we improved the previous VCM by increasing the database of 
virtual circuit components, including a picture of Arduino. Figure 2 shows examples of virtual circuit 
components that are necessary to construct virtual circuits of line-tracer robots with the microcontroller 
Arduino. Each experimenter downloaded these virtual circuit components and a template circuit board 
(breadboard) image. Experimenters used their preferred graphics editor to indicate connections of circuit 
components on a virtual circuit by placing virtual circuit components on the circuit board image. The 
experimenters indicated the virtual circuit connections by drawing colored lines on the circuit board image 
using a graphical user interface. 
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2.2 Circuit Recognition Function 

The proposed e-Learning system performs circuit recognition using image-processing techniques (pattern 
matching) that compare components in a virtual circuit with each circuit component available in the database 
of the VCM system. This function searches and recognizes circuit components connected at each node in a 
virtually constructed circuit. Checking the differences of circuit components at each node between the circuit 
diagram and the circuit virtually constructed using VCM, this system detects incorrect parts in the circuit 
constructed using VCM and informs the user of such parts. 



Figure 2. Virtual circuit components: (a) picture of Arduino, (b) MOS-FET, (c) optical sensor (photoreflector), (d) 
resistor, (e) DC motor, (f) 9 V battery, and (g) breadboard. 


2.3 Translation and Simulation Functions of Arduino Program 

Arduino is a microcontroller that is widely used for many purposes such as manufacturing, business, 
and education. The programming environment of Arduino is open source and usable with various types 
of computers and operating systems (Windows, Mac OS X, and Linux). To program Arduino, a user can 
use a programming language that is similar to general programming languages such as ANSI-C or C++. 
Therefore, learning Arduino programming for machine control and robotics is suitable for 
undergraduate and graduate students in the field of technical engineering. This paper proposes a 
function that can translate Arduino programs written by individual users into the programming language 
named Processing (http://www.processing.org/), which is robust for image processing, computer graphics, 
and animation. Processing is also open source and usable with various types of computers and operating 
systems (Windows, Mac OS X, Linux, and Android). The grammar of Processing is based on a general 
programming language such as ANSI-C or C++. Therefore, translating an Arduino program to Processing is 
possible with some partial rewriting, for instance, the proposed e-Learning system includes a rewritten 
description of the signal intensity obtained from a sensor into that of the brightness of an image pixel in a 
Processing program. Based on translated Processing programs, users can check a robot’s motion on their 
monitors. This technique is important because an experimenter can learn about robot motion without 
constructing a physical robot. 

2.4 Experimental Methodology 

The proposed system was evaluated by five undergraduate students in a class at TUAT. To evaluate the 
usability of each function of the proposed e-Learning system, the students were asked to perform the 
experiments as follows: 

A) Each learner downloaded the circuit diagram of a line tracer to be constructed (Figure 3) via the 
Internet. 

B) Using the VCM function of the proposed system, each learner made the virtual circuit of a 
line-tracer robot, including optical sensors (photoreflectors). 

C) Each learner checked whether this e-Learning system indicated incorrect parts in their circuits. If 
incorrect parts were indicated, the learner corrected such parts and completed the circuit making. 

D) Each learner wrote an Arduino program to control the virtual circuit of the line tracer that was 
virtually constructed using the VCM function. The proposed system translated this Arduino program into the 
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Processing program. The learner studied how the line tracer behaved using the simulated moving image of 
the virtual robot’s motion that was provided by the proposed e-Learning system. 


3. RESULTS AND DISCUSSION 

Five undergraduate students at TUAT evaluated the proposed system through the experiments described in 
Section 2.4. Figures 4(a) and 4(b) show example circuits of line tracers that were constructed by two 
experimenters using the VMC function. As shown in Figure 4(b), terminals of one of the diodes and a 
MOS-FET are connected at incorrect nodes, and the proposed e-Learning system shows red indicators to 
inform this learner about these incorrect parts. 

Figure 5 shows the virtual construction and simulation of the circuit of a line-tracer robot. Figure 5(a) 
shows an example circuit of a line tracer with an Arduino. Figure 5(b) presents the Arduino program written 
by the same student. The translation function of the proposed system translated this Arduio program into the 
Processing program and supplied the student with the simulated moving image of the robot’s motion based 
on the Processing program. Figure 5(c) shows a frame that was captured from the simulated moving image 
based on Program (b). 


File Download 

| Line Tracer #1 j^SubrnitJ 



File Download — 

| Line Tracer #2 | Submit | 



(b) 


Figure 3. Circuit diagram of a line-tracer robot to be constructed using the proposed system: (a) circuit to control DC 
motors and (b) circuit to detect the degree of brightness (luminous intensity) using an optical sensor (photoreflector). 
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Figure 4. Circuits of line tracers constructed by two students using the VCM function: (a) correct circuit and (b) incorrect 

circuit and automated indication of incorrect parts. 
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From the results of the evaluation by five experimenters at TUAT, positive responses, which indicate the 
usefulness of the proposed system, were obtained from all students; the responses are as follows: 

• The e-Learning system was useful for learning the construction of practical circuits with 
microcontrollers and programming a microcontroller to control machines. 

• The e-Learning system was convenient and flexible because all software used in the system was open 
source. 

However, there were also a few technical suggestions for improvement, e.g., to improve the user 
friendliness of the e-Learning system, a more advanced graphics such as three-dimensional computer 
graphics should be used. 



(a) 


Program Source (Arduino): 

File: | #Ss... | Submit | 

int Motor Left = 5; 

int Motor_R i ght = 6; 

double Threshold = 400.0; 

- 

void setup(){ 

pinMode( Motor_Left, OUTPUT ); 
pinMode( Motor Right, OUTPUT ); 

} 


void loop(){ 

int leftSensor = analogRead( AO ); 
int rightSensor = analogRead( A3 ); 
delay(1 ); 


if( leftSensor < Threshold ) 

digitalWrite( Motor_Left, HIGH ); 

e 1 se 

digitalWrite( Motor_Left, LOW ); 


if( rightSensor < Threshold )<BR> 

digitalWritef Motor Right, HIGH ); 

else 

digitalWrite( Motor Right, LOW ); 



* J 


Simulation | Reset | 



(b) 


(c) 


Figure 5. Results of the virtual construction and simulation of the circuit of a line-tracer robot: (a) virtual circuit of a line 
tracer constructed using the VCM function, (b) Arduino program written by the student, and (c) one frame that was 
captured from the simulated moving image based on Program (b). 
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4. CONCLUSION 

This paper proposes a novel e-Learning system for learning the construction of electronic circuits and 
programming a microcontroller to control a robot’s motion. The proposed system comprises a system to learn 
the construction of electronic circuits in a line-tracer robot with the general-use Arduino microcontroller and 
a simulation system to learn behavior of robots. The usefulness and effectiveness of the system were verified 
by five undergraduate students in a university class. Positive responses, which pertain to the usefulness and 
efficiency of the proposed system, were obtained from all students. The steps that are necessary to practically 
implement the proposed system are as follows: 

• Improve the graphical environment, as was reported by the experimenters. 

• Enhance the e-Learning system to study more highly developed robots with various types of sensors. 
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ABSTRACT 

The purpose of this study was to research the effect of the rubric on the level of learners’ cognitive engagement and to 
explore the effectiveness of Web-based concurrent discussions experienced by learners. The participants were 20 
undergraduate students of D Women’s University and they were divided into six groups for concurrent discussions in 
Web-based environment. Three groups of 10 participants were provided with rubric I to conduct self-evaluation and the 
remaining three groups of 10 participants were provided with rubric II to conduct self-evaluation and peer evaluation. 
After concurrent discussion in Web-based environment, it was found that the degree of learners’ cognitive engagement 
using rubric II showed higher-level cognitive engagement than the other groups using rubric I. In addition, with regard to 
the level of learners’ cognitive engagement, the groups using rubric II interacted more frequently with one another than 
the other groups using rubric I. 


KEYWORDS 

Concurrent discussion, rubric, cognitive engagement, peer evaluation 


1. INTRODUCTION 

Web-based discussion is considered important in that it contributes to expanding opportunities of 
convergence through the text-based interaction and promoting critical thinking and problem solving skills, 
which are important elements to lead successful learning in STEAM education (Lim, 2005; Lim, Yeon, & 
Kim, 2011; Harasim, 1990). Besides, the contents of discussion in Web-based discussions can be modified 
because the communication is text-based (Lee & Yang, 2009) and introspective learners who were difficult to 
participate in face-to-face discussions will be more active (Koh, 2004). In addition, it has positive effects 
such as the possibility of many-to-many discussions (Harasim, 1990), reduction of the impact of non-verbal 
elements like appearance and voice (Weisskirch & Milburn, 2003), mitigation of opinion collisions by plain 
emotional expressions, and the record of the contents of the discussion (Kyungsun Hong, 1999). Despite 
these advantages, however, most of learners experience difficulty in Web-based discussions. Therefore, 
various studies regarding design strategies for Web-based learning environment to promote discussions and 
to develop support tools are required in order to obtain an intended level of learning performance effectively 
(Park, 1998; Lim, Yeon, & Kim, 2011; Harasim, 1990). 

One of strategies for overcoming the difficulties of Web-based discussions and eliciting the learning 
effect is to provide scaffoldings to support the discussion process. Among them, the rubric has been widely 
utilized as a strategy for leading discussions effectively and promoting cognitive engagement of a higher 
level such as critical thinking and problem solving skills. In addition, the evaluation area in the rubric is 
presented more precisely by describing the execution criteria in detail and it is possible for learners to 
enhance their critical thinking by evaluating objectively their and others’ opinions. In particular, it can 
promote spontaneous interaction and lead the discussion effectively by inducing the learners’ active 
participation in the discussion and providing opportunities for checking up the demonstration process with 
clear evaluation criteria (Toth, Slithers, & Lesgold. 2002). This study intended to improve the level of 
cognitive engagement of learners utilizing the rubric in the Web-based concurrent discussions by focusing on 
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the effect of these characteristics of the rubric. In addition, this study was to explore learners’ experiences in 
the progress of Web-based concurrent discussion. For this, the major research questions are as follows: 

First, what are the effects of the rubric type on the level of learners’ cognitive engagement in Web-based 
concurrent discussion? 

Second, how do the learners recognize the effectiveness of Web-based concurrent discussions using the 
rubric? 


2. METHODS 

2.1 Participants 

The participants were twenty undergraduate students in a college of education. There were female students 
who took the course titled ‘Methodology and Technology of Education.’ This is a required course for 
pre-service teachers’ training. Most of the students were juniors and had prior knowledge related to learning 
theories such as behaviorism, cognitivism, and constructivism. The students were randomly assigned to one 
of six groups. 

2.2 Procedures 

The independent variable of this study was the type of rubric and the dependent variables were the level of 
learners’ cognitive engagement and the effectiveness of Web-based concurrent discussion using the rubric. 
The research design of the study is shown in Table 1. 


Table 1. Research design 


Gl 

01 

XI 

02 

G2 

01 

X2 

02 


Gl: the group of self-evaluation 

G2: the group of self-evaluation and peer-evaluation 

XI: rubric I for self-evaluation 

X2: rubric II for self-evaluation and peer-evaluation 

Ol: pretest to measure the participants’ prior knowledge 

02: posttest to measure the effectiveness of the rubric 

To examine the effect of the rubric for Web-based concurrent discussion, this study was conducted 
through the procedure as in Table 2. 


Table 2. Research procedure 


Phase 

Procedures 

Research design 
Experimental tool development 
Group design 
Pretest 
Experiment 
Posttest 
Data analysis 

Design of the research model 
Development of test tools 
Placed in the experimental and control groups 
Check the homogeneity of the two groups 
Web-based concurrent discussion 
Measure the effectiveness of the rubric 
Analysis of the level of cognitive engagement and the effectiveness 


2.3 Research Tools 

2.3.1 Web-Based Concurrent Discussion Environment 

Web-based concurrent discussion environment was implemented using the cafe service of portal site Daunt in 
consideration of user-friendliness and accessibility for the teacher and learners (see Figure T). 
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Figure 1. Web-based concurrent discussion environment 


2.3.2 Pretests 


A pretest was used to measure the learners’ knowledge before the Web-based concurrent discussion, and the 
test consisted of ten multiple-choice questions. The questions were related to discussion topics such as 
behaviorism, cognitivism, and constructivism. The reliability coefficient of the pretest (Chronbach’s alpha) 
was .51 and identity verification results showed homogeneity between the two groups (F=.29, p=.60). 

2.3.3 Rubric 

The rubric has been provided to support self-evaluation and peer-evaluation. Based on the previous studies of 
Paul and Elder (2013), this study used the rubric to develop subjective questionnaires. The rubric used the 
same questions for both self-evaluation and peer-evaluation and a 5-point Likert scale was developed to 
measure five items of the rubric such as clarity, accuracy, relevance, logical order, and depth. 

2.3.4 Cognitive Engagement Level 

Analysis of the level of learners’ cognitive engagement in Web-based concurrent discussion used Lee’s 
(2005) cognitive engagement analysis system of online learning. It consisted of four items such as criticism, 
expansion, repetition, and unproductivity. In addition, the Discussion Analysis Tool (DAT) by Jeong (2005) 
was used in order to investigate the effect of the rubric on the flow of Web-based concurrent discussion. 

2.3.5 Effectiveness of Web-Based Concurrent Discussion 


The perceived effectiveness of Web-based concurrent discussion using the rubric was measured after the 
discussion through open-ended questions. The open-ended questions were about how the learners recognized 
the effectiveness of Web-based concurrent discussion and the rubric. 


2.4 Data Collection and Analysis 

The dependent variable analyzed in this study was the level of cognitive engagement and the independent 
variable was rubric type. Lor the comparative analysis of rubric types, '//-test was conducted by PSAW 
Statistics 18 and the significance level for statistical verification was set to .05. In addition, the DAT of Jeong 
(2005) was used to analyze interaction between the cognitive engagement level and the flow of discussion. 
Questionnaire analysis was conducted in order to explore the perceived effectiveness of Web-based 
concurrent discussion using the rubric (see Table 3). 


Table 3. Research procedure 



Data 

Data collection 

Data analysis 

The level of cognitive 

Discussion 

Conversations occurring 

X 2 -test 

engagement 


in the progress of 
discussion 

DAT analysis 

Perceived effectiveness 

Open-ended questions 

Survey after discussion 

Questionnaire analysis 
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ABSTRACT 

By combining emerging technologies with cognitive and education theories, we are capitalizing on recent findings from 
adaptive exploration and embodied learning research to address significant gaps in the education of brain sciences for 
school children and college level students. Through the development of virtual learning tools in combination with 
innovative education techniques, we are testing novel ways to train students of all ages about the function, anatomy, and 
evolutionary history of brains. The approach we are taking focuses on key brain regions that are typically taught in 
college-level introductory courses, as well as a comparative component that introduces features and specializations of 
other vertebrate brains. By adopting methods grounded in cognitive and learning science theories, we are studying how 
technology-rich platforms promote better learning and knowledge about the brain. The first phase of the project is aimed 
at transforming learning by allowing school students to digitally manipulate brain structures, identify real brain images, 
and 3D print and assemble brain models of different animals. Our aim is to determine how learning in this 
technology-rich context compares with traditional classroom learning. To test this, we will use behavioral studies that 
assess the learner’s knowledge and the efficacy of knowledge integration into long-term memory. Understanding the 
brain is a highly interdisciplinary endeavor that bridges neuroscience, psychology, education, biological and medical 
sciences. We call this approach Brain3M, because it is a combination of ( 1 ) virtual brain interfaces in the form of Mobile 
device apps (e.g., on smartphones and tablets) and web-based tools for learning; (2) Magnetic resonance images (MRI) 
that connect with mobile and web interfaces, and (3) 3D printing Models and puzzles of vertebrate brains. The goal is to 
devise technological tools for learning about brains, as well as understanding the mind through investigating learner 
experiences with the technology. The research represents a significant step in bringing together technology, learning, and 
the brain. 


KEYWORDS 

Embodied cognition, adaptive exploratory learning, 3D printing, 3D brain models, virtual brain interface 


1. INTRODUCTION 

In a typical neuroanatomy class, the learner can become overwhelmed by large numbers of terms and 
definitions. Brain structures vary in size, shape, and location; remembering the myriad of terms along with 
their functional and behavioral associations can be challenging for many students. Often, exams are passed 
after short-term intensive work, but the obtained knowledge does not become integrated into long-term 
memory. Our cyberlearning and technology-based methods are designed to create learner experiences that 
result in long-term and highly integrated knowledge representations, making neuroscience more accessible. 
Ultimately, we plan to capture the outcomes of learning in cognitive representation through both behavioral 
and neuroimaging methods. 

It is now known that self-directed, explorative learning with guidance from well-designed, flexible 
programs can result in successful learning outcomes. In particular, tools that provide learners with flexibility 
in navigating and examining the contents of a particular domain (adaptive exploration) are very effective 
(Chariker et al., 2011; Pani et al., 2013; Naaz et al., 2014). These tools can simultaneously present 
information, test comprehension, and identify learner-specific interests and difficulties. For example, when 
learning about brain structures, Pani and colleagues showed that computer-based interactive learning, where 
the learner explores animated computer graphical models of the brain, yield significant gains with better 
long-term retention and positive transfer to other related domains of knowledge. 
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Adaptive self-exploration promotes the learner’s discovery and organization of the various anatomical 
structures and their shape, location, and configuration in the brain. Our Brain3M approach delivers a 
structured representation of brain parts and functions, providing a more embodied and effective learning 
experience as compared to current textbook-based teaching methods. In the textbook-based method, concept 
learning is a symbolic formalization (i.e., “amodal” representations) stripped away from the perceptual and 
tactile details. Our Brain3M provides an alternative “multimodal” platform for students to learn in a 
technology-rich context. This may provide a comparable experience to real brain dissection, albeit in a virtual 
environment with more learner -driven activities and exploration interactions. We argue that Brain3M enables 
the learner to establish “embodied representations”, through the learner’s integration of visual, auditory, 
kinesthetic, and other sensorimotor aspects of the learning target (Barsalou, 2008; Glenberg & Kaschak, 
2002; Kemmerer, 2015). Thus, Brain3M allows one to integrate the obtained knowledge in interconnected 
conceptual representations, which is achieved through an individual’s experiences of bodily sensation, 
perception, and action in the learning environment (in this case, the technology-rich context). 

Embodied cognition theory proposes that cognition arises directly from somatosensory interactions with 
the real world, dependent on specific modalities (including auditory, visual, motoric, olfactory, tactile, and 
verbal experiences) that integrate within the system of knowledge representation. Adaptive exploration and 
embodied learning can significantly enhance learning outcomes, especially with regard to long-term memory 
representation (e.g., Pani et ah, 2013) Additionally, Johnson-Glenberg et al. (2014) found that embodied 
learning environments, involving kinesthetic, collaborative, and multimodal /sensorimotor engagement, 
produce superior student learning given the same time period, teacher, and educational content. There is 
further evidence from cognitive neuroscience that the brain’s sensorimotor regions are responsive to 
multimodal information as a result of embodied representation (Aziz-Zadeh & Damasio, 2008; Borghi, 
Glenberg, & Kaschak, 2004; Hauk et ah, 2004), which is different from the abstract amodal representation 
housed in the anterior temporal lobe (see Figure 1; Lambon Ralph, 2014). 



Figure 1. Multimodal, multisensory effects of embodied cognition (green) vs. amodal representation in the anterior 

temporal lobe (ATL; orange) 

Our project, builds on embodied cognition theories, to examine how adaptive and embodied approaches 
aid learning about brain anatomy and function. Brain3M targets embodied cognition in multiple sensory 
modalities reaching auditory, visual, and kinesthetic learners. By targeting all types of learners, our model 
helps to provide all participants with an improved capacity for long-term memory and retention. 


2. APPROACH AND METHOD 

We use a comparative approach with human and fish brains, to introduce an important evolutionary 
component to help the learner understand how neural specializations are linked to an organism’s 
environment, or behavioral needs. While the human brain is more familiar to most students, the simpler fish 
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brain is less so. A comparison between them highlights evolutionarily conserved areas, as well as regions 
specialized for aquatic life or processing alternative kinds of sensory information. The project focuses on 
specific brain areas (limbic system, cortical and subcortical regions) and their structure and function. For 
instance, within the limbic system the hippocampus is a highly developed structure in humans, and present, 
but less well defined in fish. The hippocampus is important for learning and memory and supports other 
aspects of cognition and behavior. A Brain3M teaching module is being developed to teach middle school 
students about the function, anatomy, and evolutionary context of the hippocampus (see Figure 2). 


Br«n3U: Eifitatyoumintf (Vaitauf Dp*s s*u* *»• 



Figure 2. Brain 3M interactive model website. Regions of interest can be manipulated independently or in 
combination with each other. Clicking a region accesses information about its function, anatomy & connections 

The module uses interactive quiz games that can run on tablets and iPads (e.g., www.neurogames.co.uk), 
and direct instruction, scientific demonstrations and 3D printed materials (e.g., brain puzzles). Knowledge 
gained through this module is compared to that of students exposed to traditional textbook methods. 20 key 
facts and ideas about the hippocampus are learned. Half of the material is taught with a traditional method, 
using Powerpoint and printed text, and half using self-directed methods with the Brain3M approach. 

A ‘waitlist control’ within-subject design is used to conduct the traditional instruction vs. 
technology-based learning, and different groups experience both conditions with the order counterbalanced 
(see Figure 3). Midway through the teaching module, the two learner groups switch methods; learners using 
the traditional methods now learn 10 new facts and ideas through the technology-based method, whereas the 
Brain3M learners switch to traditional methods. Johnson-Glenberg et al. (2014) and Tolentino et al. (2009) 
have successfully used this study design. All participants receive learning evaluations at two time points: 
Pre-test and Post-test. We ensure that the information learned by the two groups is as equivalent as possible. 


Brain 3M vs. Traditional learning: Within subjects design 


Pretest * 


Group 1: — 
. (target > 14 
students) 

Group 2: 

• (target > 14 
students) 


Traditional 

learning: 

Brain 3M: 

Traditional 

learning: 

Brain 3M: 

Cortex 

Lobes 

Limbic 

system 

Comparative 

Brain 3M: 

Traditional 

learning: 

Brain 3M: 

Traditional 

learning: 

Cortex 

Lobes 

Limbic 

system 

Comparative 


Posttest 0 


• Pretest and posttest will be the same to allow longitudinal comparisons. 


Figure 3. Study design 


2.1 Assessment of Learning 

Learning is assessed using semantic priming timed response quizzes, allowing us to tap into the learner’s 
knowledge representation structure. Semantic priming is an established cognitive paradigm that tests the 
relatedness of concepts in mental representation (Neely, 1991). Semantic priming is the backbone of 
structured conceptual representation according to prominent cognitive theories (e.g., Collins & Loftus, 1975). 
The learner responds to a ‘target’ (e.g., parahippocampus) after they have seen a ‘prime’ word or phrase (e.g., 
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false memory). The dependent measure that reflects the ‘priming effect’ (from the prime to the target) is the 
speed of response for identifying correct related pairs, such as “parahippocampus - false memory” compared 
to unrelated words such as “amygdala - false memory”. The faster the response, the more automatic the 
connection between the prime and the target, as is the case in an integrated knowledge representation system 
such as our mental dictionary about related words (e.g., honeycomb automatically primes bee). The semantic 
priming experiment will be programmed using the software E-Prime (Schneider et al., 2002). 

This research has implications for student learning, because it caters for individually tailored learning 
needs (e.g., those with below average spatial ability). In the future, neuroimaging experiments will help 
identify the neural basis of learning under technology-rich contexts vs. traditional classrooms. 


3. CONCLUSION 

Our research aims to (a) enable adaptive exploration and embodied learning through multimodal visual and 
tactile interaction with virtual and plastic models of the brain, and (b) designing behavioral and 
neurocognitive experiments to understand processes within the learner as they interact with technology. 
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ABSTRACT 

Conceptualization and implementation of computer supported teaching and training is currently not tailored to the 
paradigm of learner centration. Many technical solutions lack transparency and consistency regarding the supported 
learner activities. An insight into learners activities correlated to learning tasks is needed. In this paper we outline recent 
developments in university’s higher education concepts. We introduce InterLect as a tool to observe student’s activities in 
lecture content acquisition and describe first insights and questions derived using this tool in a lecture scenario. Based on 
a new understanding of activity transformation cycles, an elaborated didactical design is modeled, which allows to 
address and observe learning activities. Ideas for a valid evaluation process are derived in order to investigate efficiency 
of the digital tool within the didactical framework. 


KEYWORDS 

Learner Centred Tools, Evaluation, lecture acquisition 


1. INTRODUCTION 

Digitalization increases in all areas of live and especially in science. This has also initiated a process of 
change in academic research and teaching. Numerous developments in digital teaching and learning are 
offering new designs for academic teaching and learning. On the one hand scientists utilize existing digital 
developments for their own research and interchange with others, on the other hand scientists are involved in 
the development of new designs for teaching and learning (Donk, 2013). Focused on studies and teaching we 
describe the dynamic progress of digitalization in the last years with the changed interaction and 
communication between lecturers and students, students and students or between lectures and lectures: 
collaborative tools and new digital teaching-learning-offerings are increasing research topics. In recent years, 
this progress of broken up existing teaching and learning formats has also led in Germany to various 
approaches and discussions about digitalization on university, see e.g. the Higher Education Forum of 
Digitalization of the German Rector’s Conference. The discussion focusses the adjustment of learning 
environments related not only with focus on technical aspects. 

With this paper we take part in the current discussion of the digitalization of teaching and learning and 
therefore we present Interlect as a tool for student centered learning. 

Biggs teaching model (Biggs, 1999) describes three levels of teaching competences: the first level “What 
The Student Is?”, the second level “What The Teacher Does?”, which is defined as the improvement in 
management of learning and teaching, and the third level “What the Students Does?“. This third level 
explains a learner-centered approach that relies on observation of the learner and learning activities. We 
adopt the described activities on teaching-learning-offerings and digitalization in a transition from the second 
level “What The Teacher Does?“ to the third level “What The Students Does?”. Thus it is no longer a single 
question of motivation and autonomy of studies with digital learning and teaching. Another aspect is the 
transparence in processes between the various stakeholders and moreover the analysis of desires and needs of 
students in comparison with the goals of teachers, the used teaching methods etc. (see therefore (Biggs, 
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1999), (Hattie, 2008)). For this purpose we take into account the specific challenge of the observation of 
learning processes. Therefore we single out this specific challenge in this paper by presenting a 
learner-content-interface, the so called InterLect-Tool (Nicolay, 2015). InterLect stands for Lecture Content 
Interface. 

The reminder of the paper is structured as follows: In the following paper, we outline the motivation for 
this tool and give a specification of the existing InterLect-Tool with first outlooks to the didactical design and 
evaluation of Interlect. 


2. INTERLECT - A TOOL FOR STUDENT CENTERED LEARNING 

2.1 Description of the Lecture- Content-Interface 

In case of a lecture scenario current approaches, such as Tweedback (Vetterick, 2013) or TopHat (TopHat, 
2014), establish an interface between audience and lecturer. Classroom response systems are used to 
interactively quiz the audience by utilizing multiple choice tests or collecting questions from the audience 
using a ranked question boards. In terms of transparency concerning student's activity in relation to lecture 
content, these systems are limited due to an aggregation of feedback and limited depth in insights to student's 
content processing. 

Our approach bridges the gap between students’ activities and current lecture content. To observe the 
learner’s processing of lecture content in lecture scenarios, we introduced a system named InterLect 
(Nicolay, 2015). InterLect consists of two major parts. First, a web-based presentation toolkit for the lecturer 
that allows a straight forward presentation and sharing of lecture slides in lecture scenarios. Second, a 
web-based audience client receiving current presented lecture slides. The audience client is built to run on 
any mobile device or laptop without installation and serves as an annotation interface to current lecture 
content. Received lecture slides are displayed as scrollable list allowing an easy tracking of lecture content 
and fast lookup of bygone information. Entered annotations are displayed together with the slides and 
embody a student's lecture script. To annotate lecture content students can select a slide from the list by 
taping and perform basal marking processes described in the next paragraphs and discussed in detail in 
section 3. 

To increase the narrative density of a lecture, slides can be highlighted by the use of bookmarks. 
Bookmarks are intended to serve as anchor, highlighting specific sections of a lecture. Displayed as overlay 
to the list of shared lecture content, they extend tracking of information with a self-defined overall structure. 

To quickly mark content, we implemented a set of binary buttons. These buttons can be edited by the 
student to define a quick access to a solid set of memorizations into further learning iteration steps 
’’Important for the exam” or ’’Look into later”. Furthermore, these buttons can be used to mark content with 
conflicts such as ’’Need examples” or ’’Missing information”. 

The system allows the annotation of lecture content with notes. A note consists of plain text and can be 
extended by the use of hash-tags, denoted by a leading ”#”-symbol and references denoted by a leading ”@”- 
symbol. While the plain text allows input of loose information, a key feature is the use of hash-tags to 
associate content and annotations to distinctive keywords. These keywords are intended to represent 
associative nodes interconnecting different pieces of lecture information. References can be used to link 
additional slides to a note or attach external resources using Unified Resource Identifiers (URI). 

Shared lecture content as well as performed annotations by the audience are stored using an anonymous 
user-ID in our database. Besides these observations of processing information in lecture scenarios, the system 
allows a qualitative interrogation of performed annotations. Therefore an automatically generated 
questionnaire can be triggered, which asks students for their reason behind specific inputs. 

2.2 First Achievements 

As part of a first field test, InterLect was used in two lecture series with topics ’’Empirical Evaluation” and 
’’Cognitive Systems”, at the University of Rostock. Both series took one semester. Each of them was 
regularly visited by 10-15 students. The system was used by students without instructions on handling and 
purpose. The aim was to resolve technical issues for the evaluation described in this paper as well as to 
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collect first insights in usability and affordance of the system. Due to the low number of participants, we only 
collected rough first insights in affordance and usage. 

Students showed a very passive utilization of InterLect. At the first lecture of both series the system was 
introduced as optional tool to receive and annotate lecture slides in order to have a personalized script. In 
parallel lecture slides have been available via common channels, such as the Universities LMS StudIP 
(Bohnsack, 2014). Nevertheless, all attending participants to a lecture connected to the system during the 
lessons to receive and lookup presented slides on their devices. Observations showed, only 2-4 students 
where actively taking notes during the lessons. All note taking students used InterLect to write and attach 
their annotations to current lecture content. 

A first look into collected notes revealed a primary utilization of free note text and hash -tags. Students did 
neither bookmark sections of the lecture nor mark content with later processing goals or conflicts. Even 
though InterLect suggests formerly used hash-tags for reuse and students used semantically similar tags, such 
as ’’measuring”, ’’measurement”, and ’’measuring-instrument”, we observed students intuitively did not reuse 
formerly used tags to link different pieces of lecture content to one associative keyword. To verify 
comprehension and mapping competence of students, exercises where used to enquire students to build 
mental maps about learned lecture content. All students performed this task successfully. 

2.3 Didactical Design 

In order to support learning activities by a didactical design we focus on a bottom-up process of planning 
learning processes to focus on the needs and acceptance of learners. (Sahl, 2015) Those processes should go 
beyond transfer and documentation of content within a lesson. Our didactical design have to include all 
activities - from individual works to lessons - and shall adapt to the affordances of any environment relevant 
to this individual. Bottom-up learning takes place within scenarios of (inter-) action where learners have to 
apply, reflect and develop their individual skills to solve problems and fulfill tasks. It is based on the 
individual learning level of a subject, combining formal and informal aspects to a complex behavior and 
developing dynamically within the scope of needs a learner has and environmental affordances. (Laurillard, 
2012 ) 

Against the backdrop of these prerequisites lecture content reception has to be described as an active and 
generative practice of searching for inter-relations between the situational constraints a learner tries to cope 
with and material stimuli from a university course or other media of teaching. Reception of lecture content is 
about building focus and emphasizing areas of interest within given material depending on individual coping 
with situational affordances. Therefore we have to establish a situational framework (fig. 1) in order to 
explore how and why students utilize InterLect as a means to run lecture content processing. This framework 
allows predicting requirements, students have to fulfill within learning tasks. Furthermore it shall describe a 
unified set of steps, which constitute task-solving processes. 

A situational framework of learning is no static entity. According to Activity Theory (Engestrom, 2001) 
(Wansga 2011) (Nouri and Cerratto-Pargman 2015) situated learning takes place where learners run through 
transformation processes in order to fulfill tasks. These processes, described by Engestrom, are accelerated 
by specifications and contradictions, which arise from coordination between the task and individual abilities 
of a learner. We plan to apply at least three different, interrelated task sets, which develop from knowledge 
transfer to knowledge elaboration. These task sets will be featured by different teaching/learning 
arrangements. In result, transformation processes will be iterated and enhanced at different stages with 
gradually increasing learner autonomy and responsibility. InterLect will be applied throughout the whole 
process. Observation and measurement of InterLect usage shall clarify to what extend options like coding, 
linking and annotation of content support learners to align different modes of content processing with 
situational constraints they try to cope with. 
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evaluation aspect: transparency in learning processes by utilization of InterLect 


Figure 1. learner-centered application of teaching technologies using the example of InterLect 


3. CONCLUSION AND OUTLOOK 

First implementation steps proved viability and practicability of InterLect. Analysis of the first experiences 
revealed decisive benefits that can be expected from a user-content-interface within a learner-centered 
approach on learning. So far content acquisition relies on situated learning processes, which modify the way, 
learners adapt their treatment of learning content. A deeper understanding into the way a 
user-content-interface supports learner activities implies a comprehensive analysis and design of learning 
processes. Based on activity theory, a didactical framework to implement and evaluate digital support of 
learner-centered teaching has been developed. That supplies crucial prerequisites to precisely address and 
observe learning activities within a digitally promoted learning complex. 
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In the course of the development of InterLect we focus a systematic evaluation of the tool. Basing on the 
given analysis of the didactical design, we have to develop an evaluation design. For a longitudinal analysis 
qualitative and quantitative methods have to be considered, to evaluate the acceptance of the tool InterLect. 
Thereby we also have to focus on the benefits for learners and teachers in using InterLect. 
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ABSTRACT 

The paper presents a meaningful discovery learning environment called “cache -cache comparison” for a personalized 
learning support system. The processing of seeking hidden relations or concepts in “cache-cache comparison” is 
intended to encourage learners to actively locate new knowledge in their knowledge framework and check the logical 
consistency of their ideas for clearing up misunderstandings. This active engagement is also expected to prevent a) the 
emergence of cognitive overload and b) the decreasing of the curiosity and willingness to explore as in the meaning 
reception learning environment. 
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1. INTRODUCTION 

Evidence from diverse sources of researches suggests that knowledge gets incorporated into human brain 
more effectively when it is organized in hierarchical frameworks. Learning approaches that facilitate this 
kind of organization significantly increase the learning capability of learners (Bransford et al., 1999; Tsien, 
2007). Ausubel's learning psychology theories (Ausubel, 1963; 1968; Ausubel et al., 1978) define this 
effective assimilation of new knowledge into existing knowledge framework as the achievement of 
“meaningful learning”. Therefore, how to help learners to efficiently develop their conceptual framework 
becomes the main issue for fostering meaning learning in e-learning systems. 

However, most of e-learning systems organize the knowledge of a curriculum in a tree structure, based on 
the textbook chapters or based on the arrangement of classes. This kind of tree structure obscure the relations 
between knowledge points (KPs, a KP is defined as “a minimum unit which can independently describe the 
information of one knowledge and be understood by its own expression or be acquired by practices” in this 
research). Especially for those KPs which belong to different branches, it is difficult to emphasize their 
relations (such as similarities and contrasts) in teaching process. Furthermore, it is also difficult to identify 
the relevant knowledge a learner possesses before and after a learning activity. To resolve these difficulties, 
an ontology-based customizable learning support system (CLLSS) which uses a hierarchical map structure to 
manage the KPs of a curriculum has developed to foster meaningful learning (Wang et al., 2014). 

Although in the latest version CLLSS 2.0 already provides visualization supports for the emphasis of the 
relations between KPs and enables learners to easily locate needed learning material addressing relations. 
However, CLLSS 2.0 simply provides a meaningful reception learning environment which directly displays 
the relation map created by experts. It could easily lead to passive learning and lower the learner’s willing to 
explore new knowledge. Therefore, to encourage active engagement, we present a timely guided discovering 
learning environment called “cache-cache comparison”. “Cache-cache comparison” encourages the learner to 
detect hidden relations between relevant KPs or hidden acquired KPs from given relevant KPs and relations 
through reflecting the attributes of acquired knowledge. If learners keep making mistakes during the task 
system will provides hints to help them rethink and get closer to the correct answer. The processing of 
seeking hidden relations or concepts is intended to encourage learners to locate new knowledge in their 
knowledge structure and restructure existing knowledge; meanwhile the iterative procedure of confirmation 
and modification in their own relation map ensure that they check the logical consistency of their ideas and 
clear up misunderstandings. 
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2. THE PREVIOUS RELATED WORK 

To facilitate meaningful learning, learning materials should be well constructed with concepts presented with 
examples related to leaner’s prior knowledge. On the other hand, the learner must possess a good attitude and 
motivation for the construction of knowledge framework by comparing new knowledge with acquired 
knowledge. The meaningful learning will impeded by low attitude and motivation even the knowledge in 
learning materials are well-organized. Although not only instructional strategies but also evaluation strategies 
which emphasize and encourage learners to relate relevant knowledge can foster motivation improvement, 
this paper focuses on the instruction environment in the e-learning system from system design perspective. 

In previous work, from the perspective of learners' knowledge structure, we developed a meaningful 
reception learning environment(Wang et al., 2013) by organizing KPs in a map structure and clarifying all 
the relations between KPs. In this meaningful reception learning environment (as shown in Fig. 1), whenever 
learners are studying a new KP, they will be presented with the relations between the new KP and relevance 
KPs especially the acquired ones to assist them to reach correctly understanding. 

It is found that the experimental students who learned with this environment achieved significantly better 
learning achievement than those who just did self-study with textbooks after studying the same target 
Japanese grammar contents (Wang et al., 2014). This suggested that the new KP can easily be understood and 
remembered through this visualization support. However, students reported that they felt pressure and 
disturbed when more than 4 related KPs are shown at one time. In other words, from cognitive load point of 
view, the e-learning environment need to avoid directly giving a big number of information at one time. 
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Figure 1. The relation map in CLLSS 2.0 

From learning attitude and motivation perspectives, the leaner data before and after studying target 
grammar contents with the support of CLLSS 2.0, were also analyzed (Wang and Mentori, 2015). By 
considering learning attitude and motivation before the learning activity as individual difference variables, 
we found that learners with high level of attitude/motivation perceived a greater effect on developing the 
habit of “learning from the comparison of related knowledge” and felt more satisfied with the learning mode 
in CLLSS environment. Moreover, compared to learner with low level of attitude towards Japanese grammar 
before the activity, learners with high level of attitude perceived significantly lower mental effort while 
studying with CLLSS and achieved better achievement on the grammar test after. These results confirm that 
the learning attitude and motivation is an essential condition for meaningful learning. 
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The analysis of learner data also suggest that not only learners' attitude towards Japanese grammar 
learning, but also their motivation toward Japanese language learning improved after studying with the 
system. However, in the interview after the experiment, 19 out of 60 participants expressed that, since the 
system already provides lots of related knowledge they don’t have the urge to actively search more by 
themselves. Furthermore, for the items about the curiosity and willingness to explore more related knowledge 
on the attitude/motivation questionnaire, 8 students even were found with post_rankings slightly lower than 
pre_rankings. The occurrence of this phenomenon is mainly because the version 2.0 of CLLSS simple 
directly displays all the information of related concepts and relations. The participants made the comparison 
between concepts in a passive reception way. This kind of passive learning lowers learners’ willingness to 
explore. This reminder us that CLLSS needed to be modified to encourage learners to actively engage in the 
construction of relation map. 

Therefore, for solving the potential problems in both cognitive load and attitude/motivation, “cache-cache 
comparison” environment is presented for CLLSS 3.0. This environment, which hides some relations or 
acquired KPs and guide the learners to actively recall their prior knowledge to design their own relation map 
before comparison with the relations map of experts, is intended to lower the cognitive load and encourage 
learners’ active engagement. 


3. CACHE-CACHE COMPARISON 

3.1 Why “Cache-cache”? 

The word of “cache”, which originally comes from French, means “to hide” or “a hidden place” (Chiaki Itoh 
and Seiji Fujino, 2014). “Cache memory” as a high speed storage device made the word “cache” well known. 
In French, some words are formed by combining the same word twice, such as “bonbon” or “cache-cache”. 
The meaning of the compound word is different from the original simple word. The origin of this kind of 
words could be that, it seems quite impressive for little children that a short word is pronounced twice and 
this impression make the word easier to be remembered. The French word “Cache-cache” means “hide and 
seek” in English. It is one of the popular children games in which one or more player as seekers tries / try to 
find several hidden players. Although there are slightly different versions of this game, with a simple rule 
most of children can immediately participate in it and enjoy the game. Ishare Company from Tokyo has 
conducted an Internet -based survey to investigate the response to “What is the game that make you feel really 
enjoyed in your childhood?” 458 Japanese (male 54.6%, female 45.4 %,) with the age from 20 to 40 
participated in this survey. The top ranking answer was “cache-cache” with 19.7% of supporters. 

Therefore, taking advantage of this familiarity, we propose to apply the word of “cache-cache” to 
represent a process of “hiding and seeking” similar to the kids’ game. As an adult, we sometimes consciously 
or unconsciously do “hiding” and “seeking” not only in daily life but also in research activities. For instance, 
in parallelism of numerical algorithms, programs were designed to firstly hide some parts that cannot be 
parallelize and then, after achieving the objective of parallelism, those parts will be found out again to 
complete the rest of computation. Accordingly, a “cache-cache balance” technique has been presented 
(Chiaki Itoh et al, 2014; Seiji Fujino et al, 2014). 

From learning support perspective, directly presenting too many information related to a new knowledge 
will create a high level of cognitive load. As mentioned in Section 3, this phenomenon was also revealed in 
one of our previous work (Wang J.Y. et al, 2014). Therefore, we suggest to hide some parts of the 
information at the first stage of learning, and then encourage learners to actively detect them in the second 
stage. This process involving discovering learning is defined as “cache-cache comparison” in this research. 

3.2 Discovering Learning 

“Discovery learning is an inquiry-based, constructivist learning theory that takes place in problem solving 
situations where the learner draws on his or her own past experience and existing knowledge to discover facts 
and relations and new truths to be learned” (Bruner). Bruner (1961) stated that students are more likely to 
remember knowledge discovered on their own in contrast to those taught directly in reception instruction. A 
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learner experiences her/his individual discovery process by applying exiting knowledge to solving problems. 
This process which encourages active engagement, can foster the development of creativity and problem 
solving skills, and promote learning motivation. However, many researchers (Mayer, 2004; Alfteri et al., 
2011) also cautioned that unassisted discovering learning lacking of necessary prior knowledge and guidance 
may easily lead to misconception and cause extra cognitive overload. Timely guidance is needed in discover 
learning for learners to avoid confusing or frustrating (Kirschner et al., 2006). Learners need to gain 
confidence that they have the ability to complete the task by providing indispensable knowledge; on the other 
hand, when confronted with failures they also need to be motivated to learn from mistakes to get closer to the 
truths. 


3.3 Meaningful Discovery Learning Supported by “Cache-Cache Comparison” 


i Learning environment in CLLSS 2.0: 


Directly display relation map 


▼ 


Meaningful reception learning 


"Cache-cache comparison" environment in CLLSS 3.0 ! 

1. Encourage learners 
to detect hidden relations/KPs 
y 2. Provide timely guidance 


Meaningful discovery learning 


^easily cause 


Passive learning 


lower 


encourage 


Active engagement 


improved or at least maintain 


The willing to explore 


is positively related to 


Learning attitude and motivation 


Figure 2. Differences between two versions of meaningful learning environment 

To encourage active engagement in meaningful learning, we present “ cache-cache comparison” 
approach by integrating with discovery learning. Considering KPs and relations as the building blocks for 
relation map of a course, the system hides several blocks in the relation map of experts and guide learners to 
seek and discover those blocks. Learners are engaged in an active learning process when they struggling to 
complete the relation map. This active engagement is expected to improve or at least maintain learners’ 
willing to explore and accordingly improve their learning attitude and motivation. Fig. 2 illustrates the 
comparison between “cache-cache comparison” environment in CLLSS 3.0 and the meaningful reception 
learning environment in CLLSS 2.0 from learning attitude and motivation point of view. 

To lower the confusion and cognitive pressure, from the uncompleted map learners are enable to easily 
access the explanations and practices addressing each displayed KP or relation. Learners have to ruminate on 
their features to determine what the missing KPs or linking relations are. Timely guidance will be provided 
for learners to elicit positive responses toward finding correct answers. While interacting with “cache-cache 
comparison” environment, learners need to draw on his existing knowledge and compare with the new 
knowledge in order to find the missing blocks. This process fosters learners to locate new knowledge in their 
knowledge framework with relations discovered by themselves and avoids the occurrence of misconceptions. 
The visualization interface of “cache-cache comparison” which can support the learners to actively build up 
their knowledge framework are still under development now. 
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4. CONCLUSION AND FURTHER WORK 

This paper presents a “cache-cache comparison” environment in a learning support system which encourage 
learners to detect hidden relations between relevant KPs or hidden acquired KPs from displayed relevant KPs 
and relations whenever they study a new KP. This learning environment integrates the advantages of 
meaning learning and discovery learning by providing visualization support and timely guidance to prevent 
misconceptions and lower cognitive load. “Cache-cache comparison”, intended to support the effective 
construction of learners’ knowledge framework, is also expected to prevent the decreasing of the curiosity 
and willingness to explore related knowledge by encouraging active engagement. 

After the development of “cache-cache comparison” environment, further evaluation will be conducted 
to compare the learning performance differences between learners who studied with “ cache-cache 
comparison” and those who studied with the meaning reception learning environment in CLLSS 2.0, 
especially in the attitude and motivation aspects. 
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ABSTRACT 

Students today share a unique characteristic in that they have all been raised in a digital era. Students from all 
socioeconomic levels have been impacted by digital media in one way or another, and many yearn to be engaged through 
the use of digital technologies in the classroom. One such digital project has sought to aid in meeting the need to develop 
such skills in preservice teachers and subsequently transfer these skills to future students. This project has been coined 
with the term App Smashing. App Smashing is the process of developing content on multiple digital applications and then 
integrating or "smashing" them together in order to create a richer, innovative digital product (Kulowiec 2013). 
Combining digital apps can be utilized for the process of research, note taking, understanding, creating, and sharing in 
any grade level classroom or subject matter. This paper seeks to further define the process of App Smashing and examine 
relevant literature. Lastly, one example of a project that utilizes App Smashing will be described. This project has been 
implemented in a teacher education course in order to educate preservice teachers about how to actually design a 
student-centered, project-based assignment and then reflect about the process upon completion. 

KEYWORDS 

Technology Integration, Teacher Education, Student -Centered Projects 


1. INTRODUCTION 

In the current and rapidly changing, digitally saturated environment, educators find themselves constantly 
seeking new ways to teach and engage students in “an ever-changing technology context and... who no 
longer process information primarily in a sequential manner” (Lambert & Cuper 2008, p. 264). According to 
Roa and Skouge, technology tools that are used in creative ways have the possibility to become a simple, 
interactive multimedia-authoring environment for classroom learners (2015). Moylan emphasizes that 
project-based learning encourages students to ‘learn by doing’ by completing hands-on projects (2008). 

Lambert and Cuper state that not only do these challenges affect classroom teachers, but it also makes 
teacher preparation “increasingly important and increasingly challenging as teacher educators seek new ways 
to integrate 21 st -century skills, nonlinear skills, and digital-age reflections into coursework” (p. 265). Many 
more recent technologies focus on “creating communities in which people come together to collaborate to 
learn and build knowledge. This is now influencing the way we communicate/collaborate within the 
framework of the teacher education programs. . .” (Mach 2013, p. 2508). 

One such digital project has sought to aid in meeting the need to develop such skills in preservice teachers 
and subsequently transfer these skills to future students. This project has been coined with the term App 
Smashing. This paper seeks to define the process of App Smashing and examine relevant literature. Lastly, 
one example of a project that utilizes App Smashing will be described that has been implemented in a teacher 
education course. The assignment seeks to educate preservice teachers about how to design a 
student-centered, project-based assignment and then reflect about the process upon completion. 

1.1 What is App Smashing? 

Although not really a new concept in the world of multimedia, Kulowiec introduced the term App Smashing, 
and defines it as incorporating or embedding multiple applications together in order to accomplish a major 
learning task or project (2013). Young (2014) states that App Smashing grants a student the freedom, through 
a plethora of choices, to create exactly the type of product they can envision. Some technology projects 
assigned in the classroom cannot be completed using just one application; some applications require using 
another application to fill in the gaps to complete a project (Kulowiec 2013). Students can complete App 
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Smashing projects individually or in cooperative groups. According to Young, App Smashing utilizes the 
media attributes characteristic of each application; one application’s distinctive characteristic may include 
integrating text and photos, while another consolidates sound with video (2014). 

In order to successfully utilize an App Smashing project. Young suggests that both teacher and students 
need to consider what will be assessed in order to produce a final, appropriate product, as well as identifying 
learning goals and outcomes with alignment to the App Smashing project (2014). Roa and Skouge (2015) 
proffer that “with its ability to combine and display media elements - such as photos, video, audio, and text - 
presentation software can be used purposefully to transform media into tools of learning and engagement for 
young children” (p. 103). Further, Kulowiec (2013) notes that an App Smashing project could even be used 
in a project -based assignment that takes place over the course of an entire semester or academic year. 

1.2 Best Practices for App Smashing 

The App Smashing process can also expose the student to project-based learning (PBL) opportunities, which 
engages students, and allows them to learn in at all six levels of Bloom’s Taxonomy; Bloom’s Taxonomy 
includes: knowledge, comprehension, application, analysis, synthesis, and evaluation (Moylan 2008). 
According to Moylan, “PBL has been identified as a key methodology for closing the gap between current 
student learning and developing the necessary 21 st century knowledge and skills” (2008, p. 287). Further, 
PBL with multimedia or through App Smashing “provides the benefit of allowing students to use multimedia 
to construct their own meanings. . .Researching a topic, evaluating resources, synthesizing core concepts, 
selecting appropriate media, and creating a product can involve students in higher order thinking and enhance 
understanding of the topic (Royer & Royer 2012, p. 2326). PBL allows students to be their own teacher; 
students build a relationship with the projects they create, which encourages them to use their imagination 
(Moylan 2008). Classroom models and practices, suggested by Price (2011), include making sense of ideas 
and holding students accountable for generating and evaluating ideas. 

The App Smashing process can be used with students of all grade levels and subjects because the variety 
and number of applications available can be combined or “smashed” in such a way that they meet the 
instructional needs of all educators (Young 2014). Young also states that App Smashing encourages student 
excitement and engagement about creating their own project, demonstrating their individual exceptionality, 
as well as combining products using familiar applications (2014). “App Smashing projects encourage 
collaboration and innovation - skills essential for success in the 21 st century (Young 2014, p. 14). Moylan 
states, “PBL is especially effective when supported by educational technology, while blended with service 
learning (learning outside of the classroom), and provides an exceptional PBL educational opportunity” 

(2008, p. 288). 

According to Rao and Skouge, the process of App Smashing or producing multimedia projects meets 
criteria for the Universal Design for Learning (UDL), which includes three main principles: provide multiple 
means of representation; provide multiple means of expression; and provide multiple means of engagement 
(Rao & Skouge 2015, p. 104). Young suggests after determining which goals or objectives are to be 
measured, various apps can function for different levels of higher order thinking skills (2014). For example. 
Young has used applications such as Skitch and Pic Collage as a first level application to share and 
manipulate images (2014). Then, Young has included applications such as iMovie and ThingLink to smash 
all first level applications into one single presentation (2014). As a part of a fifth grade curriculum, students 
must demonstrate strong skills of analysis in which Young uses App Smashing as an assessment for literary 
and character critique (2014). 

One of the benefits of using App Smashing, as experienced by the authors, is that there are numerous 
applications that can utilized as far as subject matter; as well as how they are utilized or the function; and 
even more variety and options exist in the manner in which the applications are combined or smashed. The 
project can be as complex or as simple as the educator wishes depending on the intended learning outcomes. 
The body of this paper will detail the project design process assigned by the lead author in a teacher 
education program for preservice teachers. 


2. DESCRIPTION OF APP SMASHING PROJECT 

School-age students today have all been raised in a digital era. Students from all socioeconomic levels have 
been impacted by digital media in one way or another, and many expect to be engaged through the use of 
instructional technology in the classroom, such as App Smashing. As previously described, App Smashing is 
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the process of developing content on multiple digital applications and then integrating or "smashing" them 
together in order to create a richer, innovative digital product (Kulowiec 2013). Combining digital apps can 
be utilized for the process of research, note taking, understanding, creating, and sharing in any age level 
classroom or content area. 

2.1 Purpose and Benefits of the Project 

The project that will be described in this paper is assigned in an instructional methods course for both upper 
class undergraduate and graduate students at an American private, liberal arts university. Classes are typically 
very small and include students who major in either English, history, math, or science, art or physical 
education and are seeking an endorsement to teach in that area. The process the preservice students progress 
through in order to complete the project, which usually takes half of the semester to complete, aligns with the 
topics they are learning about in class dealing with various instructional strategies, lesson planning and lesson 
delivery. A major component of this course includes lesson design and delivery of three microteaching 
experiences. 

The purpose of the App Smashing project is to provide these preservice students with the experience of 
completing a multimedia assignment they could assign to their own students. This assignment could serve as 
a form of authentic, summative assessment and also allows the preservice teachers’ future students to express 
their knowledge of a specific content area in a nontraditional way while utilizing technology. The learning 
outcomes for this assignment are multifaceted and include some of the following experiences: 1) backwards 
lesson planning design; 2) selection of and evaluation of instructional technology tools; 3) assessment design 
and creation; 4) student-centered and problem-based learning; and 5) reflecting on instructional practice. 

From a practitioner’s stance, the lead author has found that App Smashing promotes a number of 
beneficial skills in the category of higher order thinking skills, 21 st century skills, and college and career 
skills. Fligher order thinking skills include synthesis, creativity, reflection and critical thinking. The twenty- 
first century skills category encourages student engagement, development of technology skills, 
communication, and collaboration. The college and career component provides opportunities to increase 
skill development in the areas of research and writing, project-based learning, and presentation skills. 

This list was gleaned from classroom observation and discussion, as well as post-project reflections. 
Many of the preservice teachers commented in their reflections about the benefits of learning how to use the 
various technology tools, as well as some of the other skills previously listed. Specifically, many mentioned 
that they enjoyed getting the opportunity to explore newer instructional technology tools and learning how to 
apply them in an academic environment. Lastly, a few synthesized that one of the added positives of creating 
this project was getting to see the projects of their classmates. This helped them to further understand how 
the applications can be utilized in various content areas. The resulting project also becomes a quality artifact 
that the preservice teacher can include in their professional, end-of-program ePortfolio. 

2.2 App Smashing Project Process 

For this assignment, the preservice teachers are required to select a set of specific standards-based skills that 
will be assessed for student mastery. For the sake of this course, the preservice teachers create the assignment 
individually because of the variety of disciplines that may be represented in the course. They are, however, 
informed of the collaborative nature that many of the technology applications possess, thus increasing their 
awareness of how such a project may lend itself to being a group project when utilized in their own 
classrooms. The resulting project assignment could be an extension activity from a unit test, research paper, 
or other more traditional form of assessment. Once the standards/goals have been selected, the preservice 
teachers write the behavior objectives for the assignment. The parameters and requirements for the project 
they design are based upon the behavioral objectives. 

The process for the assignment then proceeds as follows. First, the preservice teacher must choose 
between either Prezi or Padlet as their main presentation platform. This project focuses on the use of free or 
affordable applications. Animoto and Zaption will be two secondary applications the preservice teachers will 
utilize to create short digital pieces to further enrich their primary presentation. Optional applications include 
integrating a word cloud utilizing either Wordle or Tagxedo, or an illustration using Postermywall. 

These applications, plus other forms of media like images and text, will be integrated or “smashed” into 
the main platform, thus the name App Smashing. 

Before the preservice teachers begin work on completing their student-centered project and following 
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their identification of standards-based skills and objectives, they must first design the project (following the 
professor’s requirements) and choose evaluation criteria upon which they plan to assess student work in the 
form of a rubric. The preservice teachers submit and receive feedback on each step of this project before they 
begin working on the technology components. This also promotes involvement in the backwards design 
process. Short tutorials are embedded in class meetings for each application, as well as how to integrate the 
various pieces of the project. The preservice teachers are required to investigate and learn how to operate or 
navigate each application on their own time. The professor, however, is always available for questions and 
guidance with regards to the applications. 

Because it is important for preservice teachers to examine their own learning and the design process for 
this assignment, they are required to write a two-page paper describing the parameters of their student 
assignment, as well as a critique evaluating the usefulness of their chosen platform and the applications as 
instructional technology tools in the classroom. They must detail how they might improve the assignment and 
if they feel the chosen applications were useful for student learning in the appropriate content area. 

Assessment criteria for the preservice teacher’s final product includes the following: 1) a title is present, 
the purpose is clearly evident, and is linked to a specific standard; 2) incorporates at least ten forms of media, 
including an Animoto video and Zaption tour, as well as other images, sounds, and videos; 3) the Animoto 
video is at least two-minutes long and the Zaption tour contains at least four questions; 4) the ten forms of 
media have a relationship and these connections are demonstrated through text or audio; and 5 ) the project 
represents quality work on behalf of the student and it is evident that adequate time was invested in creating 
it. Each criteria component is worth 20 points, and the preservice students can score exemplary, proficient; 
developing/needs improvement; or unacceptable. 


3. CONCLUSION 

In conclusion, incorporating multimedia experiences in the classroom today allows the teacher to enable “the 
student, through multimedia assessment, to utilize technology in a meaningful way” (Moylan, 2008, p. 288). 
According to Young, the process and product of App Smashing gives the student the opportunity to challenge 
himself through the creation of a sophisticated product while empowering their motivational and creativity 
skills (2014). It is the author’s intention to equip preservice teachers with this knowledge by having them 
participate and reflect upon this experience. The author continues to implement this project each semester 
with much success, but also continues to utilize new apps, as well as incorporating engaging ways to “smash” 
the apps together. Each implementation change is based upon research and preservice teacher project 
reflections. 
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ABSTRACT 

Nowadays learning, particularly the university learning, is supported with modern information and communication 
technologies. These technologies also enable electronic learning, known as eLearning, which is now firmly established at 
almost all institutions of higher learning in developed and developing countries. Moreover, at present eLearning is being 
taken over by the so-called mobile learning (m-leaming), which is possible thanks to the rapid growth of mobile devices 
such as notebooks, smartphones or tablets. In comparison with eLearning, m-leaming provides further opportunities for 
more effective learning in the sense of its wireless connections, mobility and portability, full ubiquity or instant 
information sharing. The aim of this article is to explore whether university students at the Faculty of Informatics and 
Management in Hradec Kralove are well -equipped for this new smart learning and whether they use mobile technologies 
for their studies or not. 


KEYWORDS 

eLearning; m-leaming; smart; survey. 


1. INTRODUCTION 

eLearning started to be used in the mid-1990s (Garrison, 2003). Originally, its technological component was 
preferred since it was mainly managed by IT and technical experts. However, later, emphasis was put on its 
educational value (Simonova, 2010). In this sense eLearning is usually defined although there is not one 
single definition. For the purpose of this article, the authors follow the definition provided by Wagner who 
says that eLearning is the educational process which uses information and communication technologies for 
designing courses, distributing the learning content, for teacher -learner and learner-learner communication 
and managing the whole process (Wagner, 2005). 

Obviously, eLearning brings about a lot of benefits but also drawbacks which were summarized in the 
study by Klimova and Poulova (2015). The benefits in comparison with traditional, face-to-face teaching 
include easy access to study materials; easier updating of study materials; further access to additional 
materials; individual pace, time and place of studying; almost immediate feedback; modern way of teaching; 
teacher could be absent, therefore if s/he is ill, classes are possible; chance to practice more and verify one’s 
knowledge; more opportunities for communication such as a use of discussion tools and consequently, more 
electronic consultations; support of teamwork; chance to submit assignments and their almost immediate 
evaluation; attractively and dynamism of the on-line study material; support of distance learning; higher 
motivation and stimulation for students; higher prestige of the institution; standardized tuition and teaching 
environment; cost-effectiveness for institutions; and smaller demand of computers in the traditional classes. 
On the contrary, there are certain risks which might contradict with the above mentioned statements. These 
are; a lack of personal contact; problems with technology; time-consuming and demanding for creation and 
preparation; study materials available only in an on-line form; sometimes inconclusiveness of feedback; 
absence of emotions which need to be vented; students’ reluctance to study on their own; a necessity to 
determine a ratio between the face-to-face and distance classes; problems with the guarantee of education of 
good quality; and a need to evaluate an impact on students. 

Nevertheless, at present thanks to the rapid development of wireless technologies eLearning moves to 
mobile learning (Keegan, 2002). In fact. Park, Nam & Cha (2012) see mobile learning (m-learning) as a new 
and independent part of eLearning where the education contents are handled solely by mobile technological 
devices. Table 1 below then presents paradigm shifts between eLearning and m-learning. 
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Table 1. Paradigm shifts between eLearning and m-learning 


eLearning 

Wired 

Static 

Semi-ubiquitous 
Personalized 
Providing fast feedback 
Delayed information sharing 


m-learning 

Wireless 

Mobile 

Fully ubiquitous 

Situation-based ( solving real-life tasks) 

Providing instant feedback 
Instant information sharing 

(Authors ’ own source) 


The authors of this article see m-learning as a natural expansion of eLearning, which is gaining a new 
added value by this. In this sense Chen et al. propose Black-board’s mobile learn application which is a 
personalized e-learning system that can cultivate learning abilities using a self-regulated learning assessment 
mechanism that provides immediate feedback response to students and a heteronomy mechanism that comes 
from the teacher’s reminders (Chen et al, 2013). 

The aim of this article is to discover whether university students at the Faculty of Informatics and 
Management in Hradec Kralove are well-equipped for this new smart learning and whether they use mobile 
technologies for their studies. 


2. SURVEY 

2.1 Research Questions 

Within a larger survey on the use of mobile technologies and social networks, the authors attempt to find 
answers to the following two research questions: 

1. Are students at the Faculty of Informatics and Management (FIM) of the University of Hradec 
Kralove well equipped for new m-learning? 

2. Do they use mobile technologies for their studies? 

2.2 Material and Methods 

In the winter semester of 2014-15, 317 FIM students were given online questionnaires in order to discover 
whether they are well-equipped for this new smart learning and whether they use mobile technologies for 
their studies. The research tools used were as follows: 

• online questionnaires; 

• descriptive statistical methods of processing the results of the survey; and 

• a comparison method of descriptive measures in analyzing the results of the survey. 

All students submitted the questionnaires. 159 of them were males and 158 were females, out of which 
184 (58%) respondents studied full-time while 133 (42%) of them were part-time students. The biggest group 
of the students were between 20-29 years old (194 respondents/ 61%), followed by 81 respondents (26%) 
who were under 19 years old. Then there were 28 respondents (9%) between 30-39 years old and only 13 
respondents (4%) were between 40-54 years old. 

The main fields of study of the respondents at FIM included: Applied Informatics (AI3); Information 
Management (IM3); Financial Management (FM); and Management of Tourism (MCR). 

2.3 Findings 

Within a larger survey focused on mobile devices and social networks, the respondents were asked five 
questions which were connected with those two research questions mentioned above. 
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Question 1: What technological devices do you own? 

As Fig. 1 below shows, most of the respondents (197 students/62%) own a notebook, followed by a 
smartphone (145 students/46%) and television (133 students/42%). Then 116 students (37%) have a mobile 
phone and 108 of them (34%) also have a personal computer. 



Figure 1. Respondents’ ownership of technological devices 


Question 2: What technological devices do you use for communication at school/work? 

Only in this question students have a limited number of options and could choose only four technological 
devices. Almost all respondents (297 students/94%) prefer the face-to-face communication. A considerable 
majority of the respondents (260 students/82%) also use a notebook for their communication; and more than 
half of them (206 students/65%) a smartphone. Then less than half of the respondents use a mobile phone 
(154 students/49%). See Fig. 2 below for more information. Fig. 2 also shows that there are no differences 
between men and women in their preferences for face-to-face contact and using a notebook for 
communication. Flowever, there are noticeable differences in using smartphones and mobile phones. 



Figure 2. Respondents’ communication tools at school/work 


Question 3: What technological devices do you use for your university studies? 

283 respondents (89%) exploit for their university studies their notebook. 159 respondents (50%) then use 
their smartphone and 118 respondents (37%) use their personal computer. Far fewer respondents (62 
students/20%) also use a tablet for their studies and 55 respondents (17%) a mobile phone. As it has been 
already stated in Question 2, men in comparison with women prefer smartphones while women rather use 
mobile phones. 
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Question 4: What technological devices do you use for your other studies? 

As in Question 3, students use mostly a notebook (271 respondents/86%). This is then followed by using 
a smartphone (169 respondents/53 %) and a personal computer (126 respondents/40%). A number of 
respondents also watch television (96 students/30%). Then 69 respondents (22%) also study with the help of 
their tablet and 54 respondents (17%) also use a mobile phone. See Fig. 3 below for further illustration. 



■ male 

■ female 


Figure 3. Respondents’ technological devices for other studies 

Question 5: What sources of information do you use for your university studies? 

As far as this question is concerned, 291 respondents (92%) use electronic materials from their 
Blackboard online course and almost the same number of the respondents (288 students/9 1%) attend lectures. 
In addition, 222 respondents (70%) study materials available on the Internet and 199 respondents (63%) use 
materials on university web pages. 162 respondents (51%) still go to the library to borrow the study 
materials. But as Fig. 4 indicates, the majority of them are females (106 students/65%). Then other sources of 
information used by the students are below 50% as Fig. 4 shows. 


Other sources 
Google+ 
Linkedin 
Discussion groups 
Face book 

Materials available from the... 

Wikipedia 

Study materials on university web... 

E-subjects in LMS 
I borrow textbooks from libraries 
I buy textbooks 
Personal attendance of lectures 



i male 


Figure 4. Respondents’ sources of information for their university studies 
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3. DISCUSSION 

The findings of the survey show that students are well equipped with mobile devices for their possible 
m-learning studies. This is also confirmed by other research studies ((Ozdamli and Cavus, 201 1 ) or (Cheung, 
2015)) who claim that nearly all students nowadays own mobile devices and about half of them own more 
than one. 

Although 94% of students still refer face-to-face communication, they also use a notebook and a 
smartphone. Furthermore, they use in particular these mobile technologies for their university and other 
studies. Not surprisingly, women due to their nature, tend to use mobile phones more because they are 
usually more communicative than their male counterparts. On the contrary, men who want to catch up with 
the latest technological gadgets prefer smartphones. 

In addition, the sources of information for their university studies indicate that al-most all students use 
both electronic materials and face-to-face lectures to complete their studies successfully. This is in fact a 
trend nowadays because the most common form of learning at tertiary institutions is blended learning. 
Blended learning seems to be on its rise and well established delivered methodology since more and more 
universities are becoming aware of its benefits such as greater access to students, economical use of faculty 
space, time and costs (cf. (Porter et al., 2013)). 


4. CONCLUSION 

The findings of the survey confirmed a positive attitude to the mobile devices. In fact, they are already 
integrated into daily life, they support distance, lifelong, authentic learning (EDUCUASE, 2010). And thanks 
to their other attributes such as portability, ubiquity or instant information sharing, they are very popular 
among the present (young) generation. Thus, it seems that m-learning is gradually taking over eLearning 
studies. Nevertheless, to make this m-learning really efficient and effective, the developers of mobile 
technologies should take into account the following aspects: 

• Portability : The technology is available whenever the user needs to learn. 

• Individuality : The technology can be personalized to suit the individual learner's abilities, 
knowledge and learning style, and is designed to support personal learning rather than general office 
work. 

• Unobtrusiveness: The learner can capture situations and retrieve knowledge with-out the technology 
becoming overly noticeable or imposing on the situation. 

• Availability: The learner can use the technology anywhere, to enable communication with teachers, 
experts and peers. 

• Adaptability: The technology can be adapted to the context for learning and the learner's evolving 
skills and knowledge. 

• Persistence: The learner can use the technology to manage learning throughout a lifetime, so that 
the learner's personal accumulation of resources and knowledge will be immediately accessible 
despite changes in technology. 

• Usefulness: The technology is suited to everyday needs for communication, reference, work and 
learning. 

• Usability: The technology is easily comprehended and navigated by people with no previous 
experience using it. 
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ABSTRACT 

We study tactics for writing skills development through cross-disciplinary learning in online large-scale classes, and 
particularly are interested in implementation of online collaborative activities such as peer reviewing of writing. The goal 
of our study is to carry out collaborative works efficiently via online effectively in large-scale classes with more than one 
hundred students. In this short paper we describe our recent progress of a survey of relevance between a peer reviewing 
activity and students’ characteristics with the perception obtained from investigation of earlier studies regarding the 
grouping method. We conduct questionnaire surveys regarding the learning style, students’ characteristics and 
adaptability of peer activities, and recognize the perception can adapt to most of students we supposed. Also from the 
perception if we employ an online collaborative activity in the large-scale classes by using our supporting tool, we 
consider that it is better to group students arbitrary under a certain condition. We propose a grouping method in which a 
student pairs with peers who submit his/hers assessment report around the same time as the student hand in own report. 
We think the grouping may powerfully act the group formation in large-scale classes we supposed, because the grouping 
need not to demand any additional task for instructors such as preliminary survey of students’ characteristics or learning 
styles widely used in any collaborative learning. 


KEYWORDS 

Computer-Supported Collaborative Learning, Learners’ Characteristics, Group Formation for Peer Activities 


1. INTRODUCTION 

As is well known collaborative learning activities make a significant contribution to progress of students’ 
deep learning (Beetham and Sharpe 2013). Particularly, it is considered that reviewing or grading among 
peers whether blind or un-blind plays a significant role of progress in learning of technical writing skills and 
thinking (Boud, et al. 2001). It unfortunately has specific difficulty to carry out successfully, because there 
are troublesome tasks for instructors, for example if we conduct blind peer grading, we need to deliver a 
submitted report by a student to blind peers, then back marks with comments from the peers to the student. 
Also the activities have subliminally troubling aspects. If we make a group composed students who are 
dislike or strange collaboration among others, then a collaborative activity may not perform as intended. 

There are many researches of computer-supported collaborative learning (e.g., Wessner and Pfister 2001). 
It supports not only to save the troublesome task, but also to be helpful for students to facilitate both 
knowledge and skill refinement, since peers more than instructors may be required for the refinement (e.g., 
Cho and MacArthur 2010). Recently many instructors may recognize that the computer-supported peer 
activities provide an alternative to the traditional approach of practices with feedback from an instructor or a 
teaching assistant. In fact, a computer-based grading with peers has been practiced even in some large-scale 
writing classes of MOOCs (Massive Open Online Courses, Balfour 2013). There also are reports for student 
grouping in collaborative learning (e.g., Alfonseca et al. 2006). The reports discuss relationships between 
students’ performance in collaboration and the composition of a group depending on students’ characteristics 
including learning style (Zurita et al. 2005, Deibel 2005). Most of them are based on empirical practices or 


347 


ISBN: 978-989-8533-43-2 ©2015 


heuristics testing. It is considered that a grouping method shown in a report may depend on circumstances 
such as number of students in the class, subject matter, etc., so that the method cannot always work well in 
any class. 

We study tactics for writing skills development through cross-disciplinary learning, particularly are 
interested in online large-scale classes, like MOOCs. We conduct the study in two aspects: One is 
development of a supporting tool for peer activities via online which provides relieving care of instructors to 
expense for both receiving and delivering essays among students in the large-scale classes (Matsuba et al. 
2014). The other is a survey of preliminary findings for effectively implementation of the activities focusing 
on both grouping and students’ characteristics adoptive to the collaborative works. In this short paper we 
describe our developed online collaborative activity support tool briefly, then show our recent progress of a 
survey of relevance between a peer reviewing activity and students’ characteristics. 


2. A SURVEY FOR GROUPING IN AN ONLINE LARGE-SCALE CLASS 

The goal of our study is to carry out collaborative works via online efficiently in large-scale classes with 
more than one hundred students. We conduct to design and construct a course introduced online peer works. 
It the effort to the development, we have developed a supporting tool that can automate the painful processes 
of both receiving and delivering assessment reports among classmates when we carry out peer grading 
anonymously (Matsuba et al. 2014). The tool is a module of open source learning management system 
Moodle, so instructors can install it into their Moodle freely and introduce online peer works in their class 
more easily than that with other computer-supported collaborative learning systems. We also study how well 
group members work together, and how better to group, because we would provide students the activities as 
possible as we can. 

From investigation of earlier studies regarding the grouping method we obtain some perception as 
follows; 

• Groups composed by students with similar characteristics including knowledge, skills, and interests 
tend to be better at achieving specific learning tasks, 

• Students usually tend to work peers with similar characteristics, if an instructor permits to choice group 
members by themselves, 

• Low-achieving students can learn more in a group composed by students with different (not similar) 
characteristics than in the group composed by students with similar characteristics, 

• High-achieving students can make learning outcomes equally in any group, 

• Students, beginners in particular, have difficulty in criticizing friends and perceive grades given by 
peers to be arbitrary, 

• Students worry about variations in how criteria are interpreted, distrust peers’ evaluation abilities, 

(e.g., Williams, 1992, Webb et al. 1997, Brindley and Scoffield 1998 Zurita et al. 2005). In addition, the 
research of the relationship between peer works and learning styles of Felder-Silverman (Felder and 
Silverman 1988) suggests that 

• Learning styles seem to affect the performance of the students when working together, 

• There is not any clear correlation between the dimensions of the learning styles, 

• The tendency seems to be that groups/pairs in the active/reflective and the sensing/intuitive dimensions 
can work better, 

(Alfonseca et al. 2006). Also we understand the efficient number of students in a group empirically. It is 
that three persons in a group works efficiently, because if we set up four persons in a group, then contribution 
by a person tends to be little, on the other hand, in two persons group sometime argument may not settle into 
a consensus. With the perception we consider that if we employ an online collaborative activity like blinded 
peer reviewing in the large-scale classes by using our supporting tool, it is better to group students arbitrary 
under a certain condition by three students. We often provide an assessment report submission term for a few 
weeks so that some of the students upload it in Moodle within a few days, but some of them are in deadline. 
If we impose a peer activity in the assessment, it is convenient for students to pair with peers who hand it in 
around the same time. We name such group formation is “marathon like grouping”. 
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In order to reconfirm the perception described above and study relevance of marathon like grouping, we 
conduct questionnaire surveys regarding with the learning style, students’ characteristics and adaptability of 
peer activities. The object persons are the attending students of a medium-scale class less than fifty where we 
impose a blinded peer reviewing with a rubric in an assessment, and the questionnaires in the surveys are 
referred on the results of some different researches (Felder and Silverman 1988, Kikuchi 2007, Pintrich et al. 
1991). We can recognize the perception is roughly valid. It means that, though there are some exceptional 
students, we consider the perception can adapt to most of the students. It is also shown that students with 
visual style feel available comments by peers, and having high self-efficacy for peer activities. They think 
peer activities are to be favorable learning activities for them. It is interested that we envisaged that there may 
be positive correlation between sociality and self-efficacy in peer work, but unfortunately we cannot find 
clear result for such prediction. We have an indication that we should avoid to employ un-blind peer works if 
there are many students with high scored social skills in a class, because they tend to hesitate to exchange of 
unrestrained criticism and comments. When we compare efficacy in progress of students learning between 
grouping based on students’ characteristics or learning styles and marathon like grouping, we consider that 
the big difference does not reveal whichever we employ. The questionnaire to ask the composition of his/her 
group members in both grouping implies that even if we may take the marathon like grouping, distribution of 
students in groups tend to be roughly similar to those we make groups based on students’ characteristics. 
Thus we think the marathon like grouping may powerfully act the group formation in the large-scale classes 
we supposed, because the grouping need not to demand any additional task for instructors such as 
preliminary survey of students’ characteristics or learning styles widely used in both classroom collaborative 
learning and computer-supported collaboration. It also found that many of the students have times to discuss 
or make the report of the assessment with his/her friends having similar characteristics as preparation of the 
peer work. It indicates that if we announce implementation of a blind peer work beforehand, then students 
may have twice peer activities including a preliminary work spontaneously. These are tentative consideration 
so we will try to use the grouping in a large-scale class and. continue to study about such students’ behavior 
more carefully. 


3. CONCLUDING REMARKS 

We have studied tactics for writing skills development through cross-disciplinary learning in online 
large-scale classes. We particularly are interested in implementation of online collaborative learning such as 
peer reviewing of writing. The goal of our study is to carry out collaborative works via online effectively in 
large-scale classes with more than one hundred students. We conduct the study in two aspects: One is 
development of a supporting tool for peer activities via online which provides relieving care of instructors to 
expense for both receiving and delivering essays among students in the large-scale classes. The other is a 
survey of preliminary findings for effectively implementation of the activities focusing on both grouping and 
students’ characteristics adoptive to the collaborative works. 

In this short paper we show our recent progress of a survey of relevance between a peer reviewing 
activity and students’ characteristics with the perception obtained from investigation of earlier studies 
regarding the grouping method. We show the results of questionnaire surveys regarding the learning style, 
students’ characteristics and adaptability of peer activities. We recognize the perception can adapt to most of 
students we supposed. Also from the perception if we employ an online collaborative activity in the 
large-scale classes by using our supporting tool, we consider that it is better to group students arbitrary under 
a certain condition. We propose a grouping method named marathon like grouping in which a student pairs 
with peers who submit his/hers assessment report around the same time as the student hand in own report. We 
think the marathon like grouping may powerfully act the group formation in the large-scale classes we 
supposed, because the grouping need not to demand any additional task for instructors such as preliminary 
survey of students’ characteristics or learning styles widely used in any collaborative learning. We will try to 
use the grouping in collaborative works of an online large-scale class in the next survey. 


349 



ISBN: 978-989-8533-43-2 ©2015 


REFERENCES 

Alfonseca, E., et al., 2006. The impact of learning styles on student grouping for collaborative learning: a case study, 
User Modeling and User-Adapted Interaction , Vol. 16, Issue 3-4, pp. 377-401. 

Balfour, S„ P., 2013. Assessing Writing in MOOCs: Automated Essay Scoring and Calibrated Peer Review. Research & 
Practice in Assessment, Special Issue: MOOCs and Technology, Vol. 8. 

Beetham, H. and Sharpe, R„ 2013. Rethinking Pedagogy for a Digital Age: Designing for 21st Century Learning. 
Routledge, London, UK. 

Boud, D., et al., 2001. peer learning in higher education. KOGAN PAGE, London, UK. 

Brindley, C., and Scoffield, S., 1998. Peer assessment in undergraduate programmes. Teaching in Higher Education, Vol. 
3, No. 1, pp.79-89. 

Cho, K. and MacArthur, C., 2010. Student revision with peer and expert reviewing, Learning and Instruction, Vol.20, 
pp. 328-338. 

Deibel, K„ 2005. Team formation methods for increasing interaction during in-class group work. Proceedings of the 10th 
annual SIGCSE conference on Innovation and technology in computer science education, pp. 291-295. 

Felder, R.M., and Silverman, L.K., 1988, Learning and Teaching Styles in Engineering Education, Engineering 
Education, Vol. 78, No.7, pp.674-681. 

Kikuchi, A., 2007. Kikuchi's Scale of Social Skills: 18 items (KiSS-18), Handbook of Measurements of Social Skills, 
Kawashima, Tokyo, Japan. 

Matsuba. R„ et al., 2014. Development of A Supporting System For Online Peer Activities. Proceedings of World 
Conference on Educational Media and Technology 2014, Tampere, Finland, pp. 1092-1097 

Pintrich, P. R„ et al., 1991. A Manual for the Use of the Motivated Strategies for Learning Questionnaire (MSLQ), 
National Center for Research to Improve Postsecondary Teaching and Learning, Michigan, MI, USA 

Webb, N. M„ et al., 1997. Teachers' grouping practices in fifth-grade science classrooms. Elementary School Journal, 98 
(2), 91-124. 

Wessner, M. and Pfister, H-R., 2001. Group formation in computer-supported collaborative learning. Proceedings of the 
2001 International ACM SIGGROUP Conference on Supporting Group Work, New York, NY, USA, pp. 24-31. 

Williams, E., 1992. Student attitudes towards approaches to learning and assessment. Assessment and Evaluation in 
Higher Education, Vol. 17, pp. 45-58. 

Zurita, G, et al., 2005. Dynamic Grouping in Collaborative Learning Supported by Wireless Handhelds, Educational 
Technology & Society, Vol. 8, No.3, pp. 149-161. 


350 



12th International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2015) 


ENCOURAGING USER PARTICIPATION IN BLENDED 
LEARNING: COURSE REORIENTATION 


Alea M. Fairchild 

KU Leuven Campus Brussels, Warmoesberg 26, 1000 Brussels, Belgium 


ABSTRACT 

Blended learning, structured as a combination of traditional course instruction and additional supporting multimedia 
course content, can be used in higher education for a variety of reasons. In the case study that we examine, the 
introduction of blended learning was initiated three years ago with the purpose of creating more resources for 
coach-student interaction for a heterogeneous graduate student group. However, course results from this academic year 
have shown a disengagement of the users from the course materials and a reduced number of students submitting the final 
work for the course. In this research-in-progress, we are exploring why the students have stopped engaging with the 
course materials at midterm and will try to address how the course could be reoriented to better meet its objectives. 


KEYWORDS 

Blended learning, course design, engagement, structure, orientation. 


1. INTRODUCTION 

Blended learning is an integration of face-to-face and online learning experiences; it is not just finding the 
right mix of technologies or increasing access to learning, although a secondary outcome may be increased 
efficiency and convenience for students and professor (Garrison & Kanuka 2004). There is evidence that 
blended learning has the potential to be more effective and efficient when compared to a traditional 
classroom model (Heterick & Twigg 2003, as cited in Giannousi et al. 2009); Twigg 2003). 

The specific problem this research is addressing is the case of a graduate level course where blended 
learning was introduced for purposes of both effectiveness in meeting student needs and efficiency for a 
growing programme from a scheduling perspective. Assessment of the course is done via four learning 
activities, a draft submission of the thesis proposal, and a final submission of the thesis proposal which is 
graded on a pass or fail basis. 

Through assessment, educational strategists can determine how effective their lessons are in teaching 
students the intended facts and skills (Ballera, Lukandu & Radwan 2014). Assessment designs can greatly 
influence the learning of the students. It can also be a tool for data gathering and the results gathered can help 
teachers decide on the performance of the students (Scalise & Gifford 2006). In this case, the assessment 
activities gave us insight into what level of engagement the students had with the interactive course, and at 
what point the students stopped interacting with the course materials. In compared to years prior to the 
structural change for blended learning, we have been losing effectiveness in a lack of engagement. 

In seeing both the decline in submissions of the final proposal, as well as a decline in course usage, we 
determined that the course structure as it is currently is not meeting the objectives of the course in the 
graduate programme, and that we needed to assess structurally what is not working and how we can 
reoriented the course structure to better meet the needs of the programme and the students. 

This research-in-progress paper is structured to explain the nature of the problem, highlight what 
literature and tools we will be using for the analysis, and what initial findings we have established as to how 
to rectify the situation for the coming academic year. 
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2. LITERATURE REVIEW 

The loss of motivation and engagement of students for blended course have been extensively discussed in the 
literature. Although blended learning offers various advantages in higher education settings, many studies 
have shown that blended learning also has many disadvantages. One of the biggest problems in blended 
learning is to maintain students’ engagement and motivation throughout the course (Holley & Oliver 2010). 
In order to decide which factors decide students’ engagement and motivation for blended learning, it is 
necessary to identify blended learning components. 

According to Holden and Westfall (2010), there are three main components in blended learning that affect 
students’ learning outcome and satisfaction: instructional, media, and learning environment components: 

1. Instructional Component: Used to choose the most appropriate instructional strategies to support the 
learning objectives. In other words, instructional strategies are the products of learning objectives 
and ensure the learning objectives to be attained. Therefore, instructional quality is crucial to attain 
students’ learning outcome. 

2. Learning Environment Component: A learning environment can be synchronous or asynchronous 
and aims at ensuring the most optimum use of resources to achieve the instructional goal and 
learning objectives. 

3. Media Component: Media are vehicles that deliver content of instruction. 

Reigeluth (1983) provides specific illustration of instruction design: “The discipline of instructional 
design is concerned primarily with prescribing optimal method of instruction to bring about desired changes 
in student knowledge and skill” (p.20). Instructional outcome is the way to evaluate whether the instruction 
enhances students’ learning and reaches the purpose of the course design. Instructional outcome are classified 
in three categories: 

1. The effectiveness of instruction, measured by the level of student achievement (defined in 
various methods). 

2. The efficiency of instruction, measured by the effectiveness divided by student time and/or by 
the cost of the instruction. 

3. The appeal of instruction, measured by the tendency of students wanting to continue to learn. 
(Reigeluth 1983, p.20). 


3. CASE STUDY - PROBLEM OVERVIEW 

The graduate programme in question is designed for non-economic academic graduates offering a deep 
insight into all major disciplines of business economics and management. This progamme has an incoming 
student intake from a very heterogeneous educational background in terms of both previous degree and 
nationality, and therefore starts with a preparatory term to get all of the students to the same graduate level on 
several basic subjects prior to their main course work. 

The preparatory course Research Proposal Master Thesis (RPMT) has been designed to supports the first 
phase of the completion of a master thesis. The course was developed in 2007-2008 academic year as an 
introductory course on business research with the purpose of project management for the master thesis. The 
students must submit a written research proposal where they identify a research question and draft, develop 
and complete a research proposal. Acquiring a critical research attitude is the leitmotiv throughout the course. 

The course accounts for 3 credits (ECTS) and is graded by an academic coach using a framework that the 
students have access to from the beginning of the course. Note that the master’s thesis accounts for 15 ECTS 
and is evaluated by the thesis supervisor. There are eight coaches for the course and each coach supervises 
from 25 to 30 students. Online learning materials such as documents and videos are used to transfer 
knowledge to students, as well as two actual lectures by the Professor who coordinates the course coaches. 

In the current academic year, there has been several challenges as the class size was rather large and there 
were not enough thesis topics or supervisors initially for the incoming class. We believe this may be part of 
the lack of submission problem shown in Figure 1 . 
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Figure 1. Results of grading students’ research proposal for January exam session (G = pass, NG = fail, NA = no 

submission) 

Given its preparatory role, the course is not currently achieving its objectives. Another problem is that 
many students appear to have reduced motivation for the course. The course materials (online documents, 
videos, screencast) are useful to assist students to do their RPMT submission; however, after the deadline for 
submitting their provisional research proposal at midterm, most of the students neglected the course as shown 
in Figure 2. 


Access / Date 



Figure 2. The frequency of students accessing to the course materials 


4. RESEARCH APPROACH AND METHODOLOGY 

Given the problems discussed with the course, the research questions are designed to address two issues; 

1. What are the problem) s) with instructional design of this course that may cause the disengagement? 

2. How can this course be therefore redesigned in order to achieve its objectives? 

In order to address these questions, the initial methodology has been document analysis to analyze the 
course materials (texts, website, videos, and screencasts). Merrill First Principles of Instruction is being used 
as the theoretical framework to evaluate the course and design intervention to address problems because 
many researchers have shown that First Principles of Instruction can enhance students’ learning outcome, 
engagement and satisfaction (Frick et al. 2007; Gardner 201 1). 

These are descriptions of Merrill First Principles of Instruction principles; 

• Problem-centered: Learning is promoted when learners acquire skill in the context of 
real-world problems or tasks. 

• Activation: Learning is promoted when learners activate existing knowledge and skill as a 
foundation for new skills. 

• Demonstration: Learning is promoted when learners observe a demonstration of the skill to be 
learned. 

• Application: Learning is promoted when learners apply their newly acquired skill to solve 
problems. 
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• Integration: Learning is promoted when learners reflect on, discuss, and defend their newly 
acquired skill. (Merrill 2012, p.2 1 ) 


5. CURRENT STATE OF RESEARCH 

The first step was a meeting with the course professor to go through a structured interview using the 
framework of Merrill’s First Principles of instruction. The two items which were not positive were: 

• Does the instruction require learners to use their new knowledge or skill to solve a varied sequence 
of problems or complete a varied sequence of tasks? 

• Does the instruction provide techniques that encourage learners to integrate (transfer) the new 
knowledge or skill into their everyday life? 

After structured interviews with the professor of the course and examining the course materials and 
reviewing the literature, the initial assessment using the framework of Flolden and Westfall (2010) is that the 
course encounters problems with instructional design such as the effectiveness of instruction and the appeal 
of instruction (Reigeluth 1983) which hinder the course to achieve its objective. 

The faculty members involved in the course are now working to examine how to redesign the 
instructional components, including shortening the videos and renovating the content to reframe the 
knowledge and skills provided to better resonate with the students. This will be discussed during the 
conference presentation. 
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ABSTRACT 

Nowadays, there is a consensus that inquiry-based learning contributes to developing students’ scientific literacy in 
schools. Inquiry-based teaching strategies are promoted for the development (among others) of the cognitive processes 
that cultivate problem solving (PS) competence. The build up of PS competence is a central objective for most 
compulsory education (K-12) curricula and a critical competence for both professional career readiness and effective 
citizenship. A widely accepted framework for assessing individual students’ PS competence at large scale is PISA 2012 
Problem Solving Framework (PSF). Nevertheless, PISA 2012 PS competence assessment is primarily summative 
(one-off assessment) and disconnected from the daily school science teaching practice. Within this context, the aim of 
this paper is to address this issue by proposing a framework for incorporating assessment tasks of PISA 2012 PSF to the 
various phases of an inquiry teaching model. This framework sets the ground for supporting domain specific assessment 
of students’ problem solving competence during day-to-day school science teaching practice. 


KEYWORDS 

School education, science education, inquiry-based learning, PISA 2012 problem solving framework 


1. INTRODUCTION 

Science education plays a critical role in societies’ competitiveness and economic future (Lewis & Kelly, 
2014). Developing scientific literacy in compulsory school education requires preparing students in four main 
strands, namely (Alberts, 2009): (a) to know, use, and interpret scientific explanations of the natural world, 
(b) to generate and evaluate scientific evidence and explanations, (c) to understand the nature and 
development of scientific knowledge and (d) to participate productively in scientific practices and discourse. 

Within the rich literature on this topic, there are many studies that recognize the advantage of authentic 
learning practices such as inquiry-based learning towards developing students’ scientific literacy (Crawford 
et al., 2014; Gormally et al., 2009). More specifically, inquiry is the process in which students are engaged in 
scientifically oriented questions, perform active experiments, formulate explanations from empirical 
evidence, evaluate their explanations in light of alternative explanations, and communicate and justify their 
proposed explanations (National Research Council, 2000). To this end. inquiry-based teaching models are 
recognized as appropriate teaching strategies to support deep understanding of subject domain knowledge 
and prepare students to apply this knowledge in novel real-life situations (OECD, 2013 ). Thus, to be able to 
measure (among others) the effectiveness of inquiry-based learning, assessment of students’ problem-solving 
competences is needed (Scherer & Tiemann, 2014). 

A widely accepted framework for assessing individual students’ problem solving competence is the PISA 
2012 Problem Solving Framework (PSF), which has been developed by the Organization for Economic 
Co-operation and Development (OECD) to address the need for cross-nationally comparable evidence for 
student performance on problem solving (OECD, 2013). However, PISA 2012 problem solving competence 
assessment is primarily summative (one-off assessment) and disconnected from the daily school science 
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teaching practice. Within this context, the aim of this paper is to address this issue by proposing a framework 
for incorporating assessment tasks of PISA 2012 PSF to the various phases of an inquiry teaching model. 
This framework set the ground for supporting domain specific assessment of students’ problem solving 
competence during day-to-day school science teaching practice. 


2. INQUIRY BASED LEARNING 

Inquiry-based learning is often organized into inquiry phases that together form an inquiry cycle. However, 
different variations on what is called the inquiry cycle can be found throughout the literature (Pedaste et al., 
2015). A widely used inquiry learning model is the 5E Model, which lists five inquiry phases, namely 
Engagement, Exploration. Explanation, Elaboration, and Evaluation (Bybee et al., 2006). In our work, we 
have adapted the 5E Model by considering also the inquiry cycle proposed by Bell et al. (2010). More 
specifically, according to Bell et al. (2010), the following inquiry phases have been adopted: 

■ Orienting and Asking Questions: This phase involves the presentation of the problem to be 
engaged with and aims to provoke curiosity. 

■ Hypothesis Generation and Design: This phase involves the formulation of initial hypotheses from 
the students based on their own reason and current understanding of the matter at hand. 

■ Planning and Investigation: This phase involves the collection, analysis and organization of the 
research/experimentation processes and the related tools/resources that will facilitate these. This can 
be discovered by the students or provided by the teacher. 

■ Analysis and Interpretation: During this phase, the learners engage in experimentations following 
the processes outlined in Phase 3 and utilizing the tools/resources selected in the same phase. 

■ Conclusion and Evaluation: This phase includes reflective analysis of the learners initial 
hypotheses based on the newly acquired knowledge and experience. Moreover, it aims to assist 
learners in gaining a more holistic view of the scenario problem. 


3. ORGANISING PROBLEM SOLVING ACTIVITIES IN INQUIRY 
CYCLE PHASES 

The Programme for International Student Assessment (PISA) has proposed a widely accepted framework for 
assessing individual students’ problem solving competence at large scale, namely PISA 2012 Problem 
Solving Framework (PSF). The PISA 2012 PSF defines four (4) different steps for solving a complex 
problem namely (OECD, 2013, p. 126), as follows: 

■ Exploring and understanding the problem: this step includes (a) exploring the problem situation 
(observing, interacting, searching for information and limitations) and (b) understanding the given 
information and the information discovered while interacting with the problem situation. 

■ Representing and formulating the problem: this step includes (a) selecting relevant information, 
mentally organizing and integrating with relevant prior knowledge and (b) shifting between 
representations or formulating hypotheses by identifying the relevant factors. 

■ Planning and executing the strategy for solving the problem: this step includes: (a) clarifying the 
overall goal and setting sub-goals and (b) devising a plan or strategy to reach the goal state. After 
that, in the executing phase, the plan is carried out. 

■ Monitoring and reflecting the solution: this final step includes: (a) monitoring the progress 
towards reaching the goal at each stage including checking intermediate and final results, detecting 
unexpected events, and (b) reflecting on solutions from different perspectives and critically 
evaluating assumptions and alternative solutions. 

The range of problem solving assessment tasks included in the PISA 2012 PSF allows for describing six 
levels of problem solving proficiency that can be grouped into three main categories, namely (OECD, 2014, 
p. 56-60): 

■ High Performers (Level 5 and Level 6): students at this category can: (a) develop complete, 
coherent mental models of different situations and (b) find an answer through target exploration and 
a methodical execution of multi-step plans. 
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■ Moderate Performers (Level 3 and Level 4): students at this category can: (a) control moderately 
complex devices, but not always efficiently and (b) handle multiple conditions or inter -related 
features by controlling different variables. 

■ Low Performers (Level 1 and Level 2): students at this category can: (a) answer if a single, 
specific constrain has to be taken into account and (b) partially describe the behavior of a simple, 
everyday topic. 

In order to be able to assess students’ problem solving competence (following the PISA 2012 PSF) within 
the context of inquiry-based learning, it is essential to incorporate appropriate assessment tasks in the various 
phases of the inquiry cycle (as specified in section 2). Table 1 presents our proposed framework, which 
comprises: (a) the mapping between the problem solving steps and the inquiry cycle phases (specified in 
section 2) and (b) proposed guidelines for developing assessment tasks towards assessing each of the 
problem solving steps at the different phases of the inquiry cycle. 

Table 1. Mapping between PISA 2012 problem solving steps and Inquiry Cycle Phases 


Inquiry Phases 


Orienting and Asking 
Questions 


Hypothesis Generation 
and Design 

Planning and 
Investigation 

Analysis and 
Interpretation 

Conclusion and 
Evaluation 


PISA 2012 Problem 
Solving Steps 

Exploring and 
Understanding the Problem 


Representing and 
Formulating the Problem 

Planning and Executing the 
Strategy for Solving the 
Problem 

Monitoring and Reflecting 
the Solution 


Guidelines for Preparing Assessment Tasks 

1 . Deal with the representation of the problem 

2. Deal with relevant information to understand the problem 

3. Deal with different levels of understanding of subject 
domain knowledge 

1 . Deal with the exploration of correlations and 
dependencies 

2. Deal with a precise description of the focused problem 

1 . Deal with the correct strategies of experimentation 

2. Deal with strategies of variable control 

3. Deal with strategies for data analysis 

1 . Deal with application or transfer of problem tasks 

2. Deal with possible sources of experimental errors 

3. Deal with enhancement of experimental setting 


4. CONCLUSIONS 

In this paper, we demonstrated a framework for incorporating and contextualizing within the various phases 
of an inquiry cycle, the PISA 2012 problem solving framework for assessing individual student’s problem 
solving competence. However, the use of this framework calls for taking up some future challenges, which 
could be summarized as follows 

■ Methods for assessing PS competence: different methods can be used ranging from traditional 
paper and pencil assessments (multiple choice, constructed or open response items) to student logs, 
artefacts and portfolios (Hickey et al., 2012). Additionally, in contrast to the PS competence that is 
supposed to be subject-independent, the assessment tasks should be designed, so as to be 
subject-domain oriented and embedded in the respective discipline (Scherer & Tiemann, 2014). 

■ Support science teachers for implementing effective assessment of their students’ PS 
competences: This needs to be addressed by appropriate teacher professional development 
activities, which should not be constrained only to the process of selecting suitable assessment 
methods and developing effective assessment tasks, but they must also address science teachers’ 
teaching attitudes that can be a significant barrier to the implementation of the assessment of their 
students’ PS competences (Timmers et al., 2013; Webb et al., 2013) 

It is worth mentioning that the aforementioned framework is currently being exploited by a major 
European Initiative, namely the Inspiring Science Education (ISE) Project ( http://www.inspiring-science- 
education.org/), which aims to assess students’ PS competence at large scale, namely by involving a large 
number of schools, teachers and students from 13 EU member states. More specifically, the ISE project aims 
to develop a set of digital tools that can support the authoring and delivery of technology-enhanced science 
lessons which incorporate students’ problem solving competence assessment (with appropriately designed 
multiple choice items) as part of the different inquiry phases during daily science teaching practice. 
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ABSTRACT 

In modern video games, music can come in different shapes: it can be developed on a very high compositional level, with 
sophisticated sound elements like in professional film music; it can be developed on a very coarse level, underlying 
special situations (like danger or attack); it can also be automatically generated by sound engines. However, in all these 
approaches, no active adaptation of musical elements to user activities can be found. Context dependent and adaptive 
music elements are no longer background or emotional support stmctures, but transform to information resources, which 
can be actively used by the user. Thus, they become a kind of human computer interface. In this paper, we focus on music 
as active information resource in video games and game-based learning systems. 


KEYWORDS 

Music, information, human computer interaction, auditory interface, video games. 


1. INTRODUCTION AND ANALYSIS OF GAMES 

Up to date video games and game-based learning systems are dominated by visual output and mostly rely on 
the visual interface to pass information to the player (see Nagorsnick (2012)). Even if this meets the user’s 
expectations in most cases, it does not satisfy the multimedia potential video games have reached by now. 
The possibility of playing video games without loss of information with the volume turned off is 
characteristic for the majority of them. The graphical user interface or GUI - as part of the visual interface - 
provides the player with useful information, often permanently throughout the entire game. Optionally the 
player can blend or hide parts of the information. Nowadays, video games use special processors on the 
graphic cards in the computer, to generate the computation intensive visual information, like the background 
of a virtual world, the non-player characters’ movement etc. 

In some video games the player must keep track of information at all times. This becomes even more 
important if time is a crucial game element, e.g. as part of a strategy. Often video games contain such a large 
amount of relevant information that the GUI is about to become confusing. In really bad cases, the rendered 
virtual reality is hidden behind all the information-conveying GUI-elements. The cognitive effort to keep 
track of all this is comparably high, whereas the bandwidth of cognitive input is comparably low (restricted 
to vision, excluding auditory feedback) (see for example Mousavi et. al. (1995)). Brewster (1994) points out 
that “[sound] is a useful complement to visual output”, because it can “reduce the amount the user has to 
perceive through the visual channel”. Alty (1995) argues, that the use of music in Human Computer Interface 
design should be considered. And like studies by Brewster (1994), experiments with music representing 
hierarchical menus (LePlatre, 1998) or navigation with music (Jones, 2006) show that music is well suited to 
convey information to the listener. Music as part of the auditory interface of the video game is mostly only 
used to enhance the atmosphere experienced by the player. If it would be possible to shift some relevant 
information from the GUI to the music, three effects might be reached: 

1. Take human information processing to another level (audio adding to video), potentially also allowing 
multitasking, and make the game more interesting as multiple cognitive levels are addressed. 

2. Reduction of cognitive load on the pure visual level (GUI elements can be reduced). 

3. Relieve the visual system of the player an intensify the game experience. 

For this approach, we decided to get rid of automatic music generation using Algorithmic Composition 
techniques, like e.g. in the algorithmic music generators FractMus and SoundHelix (see 
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http://www.gustavodiazjerez.com and http://www.soundhelix.com, last visited June 2015). In contrast to this, 
our approach is based on human musical compositions for certain situations, combined with an algorithm 
which steers an automatic situation adaptation and an automatic overlay of compositional parts, to achieve 
the “music” intended by the human composer as the result, instead of simple alerts in form of earcons or 
auditory icons (see e.g. Blattner ( 1989), Brewster ( 1994), Gaver ( 1986)). 

Music is a regular part of most video games. Either it is specifically written for the video game or it is 
chosen from public music. As a first research step, we made an analysis of different computer games and 
investigated the use of music there. For the first step of the investigation we selected 7 major titles, namely: 
Dragon Age 2, Mass Effect 2, Command & Conquer 3: Tiberium Wars, Dawn of War II - Retribution, Sim 
City 4, Unreal Tournament 3 and Tekken 6. All this titles are commercially successful and representative for 
their genre. We found that the main purpose of the (background) music in this games is to enhance the 
atmosphere perceived by the player (e.g. tension). To convey information to the player, the investigated 
games used pre-recorded voice messages, GUI-elements (for instance text or health bars) or graphical 
elements as part of the virtual scene. To achieve a broader view we also examined videogames with a more 
intense use of music. The games are Flower, Auditorium, Loom, Audiosurf, Guitar Hero 3 and SingStar Vol. 
1, for which music is a key element of the gameplay. And although those games’ music emerges from mere 
background music towards a more central role it neither carries exclusive information. We found that among 
all the examined video games and genres only music games (in this case Guitar Hero 3 and SingStar Vol. 1) 
established music as an active information resource. 

The next step in our research was to extract situations where music can be used as information resource. 
We chose the CRPG genre as a basis for this investigation, because it combines a large amount of different 
elements that can be found within the game, e.g. the story told and the story turns, emotions, characters, 
battles, useable skills, quests, riddles and so forth. Sometimes time is a crucial factor too. And on top of it 
CRPGs tend to make extensive use of music to fit optimally into the atmosphere. Altogether we examined 9 
video games of the CRPG genre, each 3 for a sub-genre. The Classic-CRPGs are Baldur ’s Gate II: Enhanced 
Edition, Dragon Age: Origins and Star Wars: Knights of the Old Republic II. The Action -CRPGs are Diablo 
III, Dust: An Elysian Tail and Titan Quest. Hybrid-CRPGs, that combine elements of both Classic- and 
Action-CRPGs, where picked with Dark Souls: Prepare to Die Edition, Mass Effect and The Elder Scrolls V: 
Skyrim. Some of those feature some special elements but in general they have much elements in common, 
which we called CRPG-elements. A detailed explanation of this CRPG-elements can be found in 
(Nagorsnick, 2014). With this CRPG-elements we established a foundation for our approach to make music 
an adaptive and active resource of information, since information is created by those elements. How this 
foundation is used exactly will be covered in the following section. 


2. MUSIC AS ACTIVE RESOURCE OF INFORMATION 

As noted by Collins (2007), moving smoothly from one musical piece (so called cues) to another assists in 
the continuity of a game and the illusions of gameplay. The current methods applied to video games use 
pre-recorded musical pieces that are played in a loop or cross-faded to switch between the pieces (Collins, 
2008). The problem is that “adjacent” audio files have to match in harmony and time signature (respectively 
measure) during crossfade, so they can’t cause unintended disharmony or rhythmical stumbling (see e.g. 
Reese (2012), Collins (2007), Nierhaus (2009), Shan and Chiu (2010)). We decided to take another direction 
and thus we will introduce Algorithmic Transition as branch of Algorithmic Composition. 

The main idea behind is to utilize aesthetical understanding and creativity of human composers in order to 
skip the boundaries of computable creativity and all difficulties of Algorithmic Composition techniques. To 
achieve this goal, a human musician composes a certain number of compositional parts. With these parts the 
final music can be compiled by an algorithm. For this algorithm, we need to define rules, e.g. to define, 
which parts can be played together (vertical transition) or how parts have to be merged in time (horizontal 
transition). Next the algorithm has to keep up the compositional coherence during transitions and therefore 
being able to manipulate the compositional parts the music consist of. This ability of manipulation or 
alteration of music is also the key to use music as active information resource (Nagorsnick, 2014). 
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The first concepts are music events, which we consider as auditory events in a musical context, and 
musical components. Music events can be atomic or molecular. An atomic music event is a single, sounding 
musical tone or a single rest. They are called atomic because they can’t be divided any further. A molecular 
music event, like a sounding melody, consists of at least two atomic music events. Furthermore each music 
event needs three musical components: tone, time and timbre. This means that a music event needs a defined 
pitch (e.g. C”), a temporal information that determines when and how long the event occurs (e.g. first 
semibreve in the first bar) and at last a timbre (e.g. pianissimo cello), that makes the music event sound. 

The second concepts are musical parameters, which we consider as modifiable qualities of music events. 
Like music events, we distinguish between atomic and molecular musical parameters. The former can only 
be applied to atomic music events, whereas the latter can only be applied to molecular music events. One 
example: The atomic musical parameter pitch can only be applied to a single sounding musical tone (e.g. a’). 
Pitch would make no sense if applied to a sounding melody that combines multiple pitches. In addition a 
musical parameter can affect two musical components at a max. Would it affect all three musical components 
it would be a music event per definition. 

To be able to make use of it, we need game events - the third concept. We see game events as derivations 
of the discovered CRPG-elements. They cause state changes within the virtual reality and are the source of 
the information actively conveyed to the player by music. 

To show the applicability of our approach, we chose three CRPG-elements. To be able to investigate 
music as active information resource we modelled a virtual test environment using the game engine Unity3D. 
The game events we derived from the chosen CRPG-elements were implemented in the test environment, 
whereas each game event influences the music via rules provided by a pattern language. Those rules 
determine which musical parameter is to be used for manipulation and how, e.g. (molecular) melody for 
major-to-minor change. Furthermore they define when and for how long a manipulation is done. This way we 
wanted to ensure that the compositional coherence is always kept up. Figure 1 illustrates this by taking the 
length - measured in bars - of musical themes (each triggered by another game event) into account, both for 
horizontal and vertical transitions. 


daytime-theme 


battle-theme 


daytime-theme 


imminence-theme 




bridge 


Figure 1. A schematic view for a possible battle scenario. 


To be able to make use of the manipulation of music events by musical parameters, we passed music in 
form of symbolic data to the algorithm, similar to MIDI or MusicXML. To raise our musical themes to the 
level of music events we used audio samples, representing atomic music events, e.g. a note played pianissimo 
by a cello. These atomic music events are finally combined to molecular music events conveying information 
to the player, of course depending on the actual musical themes that in turn depend on game events. 


3. CONCLUSION AND OUTLOOK 

With our virtual environment we achieved a first successful use of music as an active information resource in 
video games with the help of Algorithmic Transition. Hence all musical themes are based on a handful of 
human-composed themes. The rest is successfully created by Algorithmic Transition. The manipulation of 
music events by means of the musical parameters is working like intended. No music event causes 
disharmony or other flaws regarding compositional coherence interfered with the experienced gameplay. 
Even if the audio quality can be improved, our first approach of using Algorithmic Transition to make music 
an active information resource in video games was successful. 

In our current research, we’ll put the formal model on a firmer ground and look at the user’s performance 
with musical support (in the form of some smaller evaluations and user observations). 
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ABSTRACT 

Making construals is a practical approach to computing that was originally developed for and by computer science 
undergraduates. It is the central theme of an EU project aimed at disseminating the relevant principles to a broader 
audience. This involves bringing together technical experts in making construals and international experts in educational 
technology in organising wide-ranging learning activities. This paper, which complements the tutorial and discussion on 
making construals in (Beynon et al. 2015a, 2015b), documents issues that arise from this encounter. Its central argument 
is that making construals offers a significant new contribution to educational technology. 
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Construal; computing education; educational technology; constructionism; software development; radical empiricism 


1. MAKING CONSTRUALS 

Making construals is a new digital skill that can be seen as complementary to ‘writing computer programs’. 
The key characteristics of making construals are introduced in (Beynon et al. 2015a), a tutorial paper in 
which the use of JS-EDEN, a prototype online environment for making construals (the “MCE”), is illustrated 
by making an elementary construal of shopping activity (a “Shopping construal”). As discussed and 
illustrated in (Beynon et al. 2015a), many variants of the Shopping construal can be developed with relatively 
modest computing expertise. These include traditional educational resources that might be used to scaffold 
learning in teaching ‘life-skills’ and simple educational games. 

Making construals is the practical core of a well-established research project (Empirical Modelling 2015) 
that was first developed for - and by - computer science students at the University of Warwick. It is currently 
the theme of an EU Erasmus+ project (“CONSTRUIT!”) aimed at dissemination to a wider audience. 
Adapting resources that were originally conceived for computing specialists for this purpose has entailed a 
significant shift in emphasis. Though the technical development of the MCE and the construals described in 
(Beynon et al. 2015a, 2015b) is being led by members of the CONSTRUIT! team at Warwick, it has been 
informed through dialogue with expert international consultants in educational technology and by several 
collaborative project activities, including workshops with schoolchildren at SciFest in Joensuu, Finland and 
with schoolteachers in Athens. This abbreviated version of (Beynon et al. 2015b) summarises the diverse 
issues that have been raised in this process. These summaries do not necessarily represent a consensus view 
of the consultants, but inform an agenda for subsequent discussion. 

The focus for CONSTRUIT! activity over the first year has been on showing that comprehending and 
making construals is potentially accessible to the non -specialist. The discussion of a Shopping construal in 
(Beynon et al. 2015a) illustrates the key principles and techniques involved. Our previous experience of 
teaching undergraduate computer students to make construals (cf. (Empirical Modelling 2005)) suggests that 
novices typically begin by exploring an existing construal that is connected with their chosen topic of 
investigation, and develop their skills through making adaptations. In some cases, this leads them to take 
ownership of the construal through making their own extensions. In any event, once students have developed 
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the ability to understand and extend someone else’s construal, they have the basic competence to start 
making their personal constmals from scratch. 

As illustrated in the derivation of a simple game from the Shopping construal in (Beynon et al. 2015a), if 
we have some explicit extension, modification or specialisation of an existing construal in mind there are 
three stages in the adaptation: 

conception: without reference to the precise syntactic form and structure of their definitions, draw up a 
plan for the conceptual reorganisation of observables. This may entail re-disposing or replicating visual 
components, revising existing observables and actions, and possibly introducing new observables and 
actions. 

identification: identify the precise names and definitions of the observables and agents to be adapted 
through open-ended interaction with the construal. This activity exploits the range of viewers that are 
available in the MCE - such as the Observable- / Function- / Agent- / Symbol-Lists and Dependency 
Maps. Domain knowledge may also be helpful in guiding the search activity. 

reconfiguration: having identified the relevant observables and agents, modify their definitions 
according to the plan drawn up at the conception stage. 

The most important consideration in this process of comprehension and adaptation is that the transition 
from the initial construal to the revised construal takes place in the maker’s stream-of-thought. That is to say, 
all interactions with the construal, whatever their status (exploratory interactions, revisions, tests, 
experiments, corrections, aberrations, etc) are conducted with reference to the live construal and its real or 
imagined referent as depicted in Figure 1 . 

In practice, the above sequence of stages would not necessarily be followed strictly. Prior identification 
and exploration of observables and agents in the construal might be required in order to make a plan for 
revision. In an ambitious development there would also be many phases of the conception -identification- 
reconfiguration sequence. The MCE provides support for generating the current script (to be saved in the 
external file system) and for recording intermediate states which the maker can restore if needed. 


ENVIRONMENT 



Figure 1. Making a Digital Construal 

Figure 1 depicts the relationship between construal and referent as it is experienced by the maker as of 
one moment in time. The maker’s understanding of this relationship, which gives the construal its informal 
meaning, is built up through interaction with the construal and its referent as these evolve over time. This 
understanding incorporates expectations about the impact of an interaction that are based on knowledge 
gained from prior experiment. The maker can in principle change observables in an unconstrained fashion, 
but in practice only makes such interactions as help to establish and refine understanding of the relationship 
between the construal and its referent. Refinement will typically involve exploring the perspective of 
different agents. In the Shopping construal, the customer is not in general able to fix the price of the items for 
sale for instance - this is a decision for the shopkeeper. The integrity of the construal relies on certain 
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assumptions about the stability of the context for interaction. For instance, we would not buy perishable fruit 
by mail order, and in some cultures the price of items would be negotiable. The most appropriate semantic 
framework for the making activity in Figure 1 regards all semantic relationships as pragmatically determined 
and rooted in the connections that can be made in the personal experience of the maker. As explained in 
(Beynon et al. 2015a, 2015b), this is the fundamental tenet of William James’s radical empiricism: 

"... the parts of experience hold together from next to next by relations that are themselves parts of 
experience. The directly apprehended universe needs, in short, no extraneous trans-empirical connective 
support, but possesses in its own right a concatenated and continuous structure. ” 

The Meaning of Truth (James 1907, reprinted 1975), Preface: p.xiii. 


2. MAKING CONSTRUALS AS AN EDUCATIONAL TECHNOLOGY 

Making construals establishes a close link between how we think about a subject domain ('how we construe a 
situation') and how we might develop software to simulate behaviours and address problems drawn from that 
domain. In this respect, it represents a new way in which to realise program-like behaviours. When we write 
a program, its behaviour is prescribed in accordance with a specification. The specification need not be 
preconceived; it may be fluid and evolving. But - though it may be implicit - it is always present, if only (as 
in an agile development) as 'what the software does now'. A construal by contrast is an informal 
representation of the playground for agency within a situation as we currently conceive it. In the construal, 
there are counterparts for the patterns of agency and interaction that we encounter through experience and 
experiment. And where the interactions with a program are constrained by a user-interface that is closely 
aligned with its specification, the ways of interacting with and interpreting a construal are open-ended and 
informed by what 'makes sense' in the maker's imagination. 

Previous papers have built on these ideas to argue that making construals is a ‘radical new conception’ in 
educational technology (see e.g. (Beynon 2007)). Educationalists, having seen so much hype around the 
latest technologies, have a proper scepticism about such claims to novelty. Their wariness may reflect the 
influence of traditional computer science thinking: from a perspective informed by elementary exercises in 
programming, developing software can be conceived as ‘implementing algorithms to fulfil a preconceived 
formally specified functionality’, and there is an abstract mathematical sense in which any two correct 
implementations are equivalent. In reality, orthodox thinking about software is very ill-suited to themes such 
as constructionism (Papert 1980; Beynon & Harfield 2010), adaptable open educational resources (Wikipedia 
2015) and transformative practices in online teaching (Ross et al. 2013; Beynon & Zhu 2013). An aspiration 
for the CONSTRUIT! project is to give an account of computing that does fuller justice to these 
well-conceived and important educational initiatives and supplies a more appropriate grounding for their 
practices. 

There are many different ways in which construals can contribute to learning about a domain. Making a 
new construal of your own can be a way of learning, as can interacting with an extant construal. There is 
much affinity between the skills required for making a new construal - as outlined in the discussion of 
learning how to make construals above - and those required for exploring an extant construal. In the latter 
context, the focus is on how we learn about a domain or a referent by making or adapting construals. 

Making construals encourages the learner to adopt an unusual stance. Much technology to support 
learning is associated with bodies of knowledge that have been well -documented and formalised. The 
learner’s interactions are staged within a frame that has been carefully prepared, possibly even so that there is 
an intended ‘correct’ path to follow. The archetypal context for making construals is one in which the learner 
is invited to act with greater autonomy. Making the connection between a construal and a referent that is 
depicted in Figure 1 is a personal matter that draws on different aspects of the learner’s experience. The 
concern is not only with what the learner directly observes but with its resonance with everything else to 
which they are sensitive. And whilst the learner is led to identify observables and dependencies in the role of 
a model -builder, this is with a view to appreciating the subtlety and richness of their immediate experience of 
the domain more fully. In sharp contrast, the objective for traditional model-making is to find simplifying 
mechanisms that can be applied to the referent to serve the specific goal in hand. What is abstracted from the 
referent in this way is in effect detached from the domain. 
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In analysing how students learn to program in languages such as Lisp, Prolog, Smalltalk and Logo, three 
aspects of experience are significant: 

prior experience of implementing algorithms; 

what is being directly experienced in interacting with the tutorial environment; 
what intuitions have been acquired through experience of the subject domain. 

In evaluating the quality of the learning experience, it is the way in which these different kinds of 
experience are brought together that counts. In particular, these three ingredients of experience guide the 
learner in making the links between familiar and new concepts. 

In making constmals, the above aspects of the learner’s experience have counterparts that are respectively 
associated with the modeller’s understanding, the evolving construal and its emerging referent. A distinctive 
feature of the MCE is the degree of active control it gives over the way in which these elements are brought 
together in the learner’s experience (cf. the way in which visualisations of states and views of observables 
can be freely configured). This encourages the student to actively control and optimise the balance between 
prior and new ( "restructured") concepts. 

Learning to make construals entails a reorientation that presents a challenge of somewhat the same kind 
that is associated with adopting a declarative rather than a procedural approach to programming. Not all 
students will welcome this reorientation, but it potentially has unique value for learning. This stems from the 
emphasis on helping the learner to rethink topics that might initially appear trivial. This is particularly 
well-illustrated in the large body of project work that has been developed using the previous variants of the 
EDEN interpreter (Empirical Modelling 2006; Empirical Modelling 2013) and has yet to be replicated in the 
online MCE. By way of illustration, consider the construals of heapsorting (Beynon 2008a), elementary 
group theory (Beynon 2008b) and the Clayton Tunnel railway disaster (Chan & Harfield 2008). 


3. THE TECHNOLOGICAL PERSPECTIVE 

The relationship between making construals and other established technologies is complex. Many 
technologies appear to be addressing closely related agendas and delivering superficially similar products. 

Making construals is significant as paradigmatic for a particular way of exploiting computer-based 
technology. Where computational thinking focuses on activities for which the archetype is ‘a mind following 
a rule’ (Hodges 2002), making construals is concerned with ‘a person making sense of a situation’. The 
transition from informal personal accounts to formal objective understanding is characteristic of such sense- 
making. Deriving programs from a construal, as illustrated in (Beynon et al. 2015a), is just one instance of 
how making constmals enables this transition to take place seamlessly. In keeping with the notion that “The 
directly apprehended universe needs, in short, no extraneous trans- empirical connective support” the 
distinction between the informal and formal components in this transition is in the mind of the maker. In one 
context (cf. Figure 1): ‘this is what reliably seems to be the case’ and, in another: ‘this is what we can 
confidently deem to be the case’. 

The transition from the empirical to formal perspectives is prominent in many settings. It has a critical 
role in expert systems (Jackson 1998) and design science (Hevner et al. 2004). Working in close proximity to 
the formal perspective attracts particular interest because of aspirations for automation and AI. By 
comparison, the human-centred activities associated with making construals serve a potentially less welcome 
role - making use of computing to provoke us into thinking more deeply. 

Polya’s reflections on teaching mathematics (Polya 2014) help to highlight the unusual qualities of 
construals. As Polya observes, though mathematics is done inductively, it is taught deductively. This can 
make the subject boring, alienating the learner from the researcher. As an environment that can enable the 
learner to cope with uncertain and chaotic elements, making construals gives valuable support for exposing 
the empirical roots of formal results. In keeping with Polya’s remarks concerning the importance of reflection 
and review for learning, the history of interactions with a construal also provides an as-if-live record of the 
processes of rationalisation and problem-solving. 

The narrative of making construals is likewise closely parallel to that of design science (Hevner et al. 
2004). The learners are developing construals (“design artefacts”) and thus constantly relating their work to 
emerging demands - perhaps from their own imagination (their “local contribution” in design science terms) 
and finally, and probably throughout the process, adding to the science (“global contribution”). This is a 
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plausible narrative, but makes challenging demands where the nature of the design artefacts is concerned. In 
(Hevner et al. 2004, p.89), Hevner discusses a mode of describing the design artefact that is based on 
searching for solutions to a family of design constraints. He then observes that in general finding such a 
solution is computationally infeasible and proposes to address this by relaxing constraints. Approaches of this 
nature, which operate with formal specifications and use sophisticated algorithms, give nothing like the 
delicate control over details of design and nuances of meaning that making construals affords. As in other 
contexts, claims concerning the character and quality of methods become rhetorical in the absence of 
appropriate representations. 

The problematic aspect of incorporating the principles of making construals alongside more orthodox 
approaches is that it creates a fundamental ontological conflict. Though the maker may retain the prerogative 
to say ‘this is what we can confidently deem to be the case’ this is of no consequence if the integrity of the 
enclosing environment is predicated on the assumption that it is in some absolute sense the case. This helps 
to explain why merely adding dependency as an extra feature in an otherwise orthodox programming 
environment introduces problematic semantic issues (cf. (Roe & Beynon 2007; Tomcsanyi 2003)). 

One of the objectives for CONSTRUIT! is to enable people other than computing specialists to make 
their own software applications. This is in the spirit of Mark Hatch's Maker Movement Manifesto (Hatch 
2014), which promotes 'making' as an activity that is accessible to all. Hatch's focus is on making physical 
things, but computing and its associated technologies have a significant implicit role. As he observes: "The 
tools of making have never been cheaper, easier to use, or more powerful." ... "It may take some practice to 
get good at some kinds of making, but technology has begun to make creating easy enough that everyone can 
make.". Curiously, a better understanding of how technology for computing can support making is required if 
Hatch’s thesis is to extend to making virtual products. Making construals is helpful in this respect, as might 
be expected given the character of the applications of spreadsheets in education in (Baker & Sugden 2007). 

In the conception phase, making construals somewhat resembles the activities that have been conceived 
as ways of teaching computer science principles and concepts without using the computer (cf. ‘computer 
science unplugged’ (CS Unplugged 2015) and ‘Barefoot’ (Barefoot Computing 2015)). In effect, describing 
computing activities in non-technical human terms abstracts away the technically sophisticated apparatus that 
enables machines to manipulate and ‘interpret’ real-world observables and dependencies that a human 
interpreter takes for granted. This shifts the focus of attention from the purely technical to high-level issues 
more central to good design practice in computing. By way of illustration, consider how adapting the 
Shopping construal to allow many instances of each item (Beynon et al. 2015a) highlights a limitation in the 
representation of the stock of items - it does not allow the maker to record the identities of individual items. 


4. CONCLUSION 

This paper has argued that making construals is a practice, and an approach to software development, that is 
more accessible and intelligible for the non-specialist than coding (cf. Granger 2015). It has also argued that 
making construals is better suited to supporting learning and major themes in educational technologies than 
are the conventional methods and environments for programming. 
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ABSTRACT 

Regarding constructivism as a learning philosophy and/or a model of knowing, a person (learner or mentor) based on her/ 
his preconceptions and on personal knowings could actively participate in an interaction with another person (learner or 
mentor) in order to construct her/his personal knowledge. In this research I will analyse 'meaning construction' within 
constructivism. I will focus on a semantic loop that the learner and mentor as intentional participants move through and 
organise their personal constructed conceptions in order to construct meanings and produce their individual meaningful 
comprehensions. Subsequently, I will provide a semantic framework for analysing the meaning construction based on 
personal knowings and personal conceptions within constructivist interactions. This research could propose a new 
scheme for interpretation based on semantics and on interaction. 
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1. INTRODUCTION AND MOTIVATION 

An interaction between learners and mentors as intentional participants could exchange questions and 
answers concerning, e.g., description, specification, explanation, argumentation, analysing, justification, 
formulation, theorising etc. The multilevel agreement-oriented interactions among learners and mentors 
could be considered, be interpreted and be analysed based on the models of the underlying processes 
involved in complex human learning. As such they could be seen as a radical constructivist account of human 
cognition and comprehension. They are actually shaping a kind of ontology of human beings. They enable 
learners (and mentors) to develop their own understandings of the complex real underlying systematic 
processes, and also of themselves [Scott (2001)]. Learning based on constructivism with reference to 
Conversation Theory, which is designed by Gordon Pask, the enterprise begins with the negotiation of an 
agreement between learners and mentors to converse about a given domain and learn about some particular 
topics and skills in that domain. It could work as an explanatory, heuristic and developmental framework. For 
more detailed information see [Pask (1975), Pask (1980)]. In fact, learning based on constructivism could 
heuristically be concerned with the questions focusing on ’What is/does ’Why is/does ...?’ and ’How is/ 
does ...?’. A person whose insights are based on her/his pre- structural knowledge, personal knowings and 
preconceptions may ask these heuristic questions and ask the interlocutor to produce some heuristic answers 
or some modified heuristic questions. What could be offered by learning based on constructivist interaction is 
a framework for thought and a semantic model to account for the emergence of the domain of human 
conceptual knowledge. As an abstract model, it is able to explain how the interactions lead humans to 
construct personal knowledge. In this framework the learner (mentor) manages to construct her/his personal 
knowledge within interaction. Consequently, s(he) gains an opportunity to attain a deeper personal 
understanding and greater motivation. According to constructivism, a learner is highly concerned with active 
‘creation’ of personal mental structures. Constructivism requires negotiation of ’meaning’ and ’reflection’ of 
prior and new knowledge. Jean Piaget, the originator of constructivism, argued that all learning was mediated 
by the construction of ‘mental objects’ that he called schemata. For Piaget, schemata first emerge as concrete 
actions and then gradually develop into more abstract and conceptual mental entities [Husen and 
Postlethwaite (1989), Spiro et al. (1991), McGaw and Peterson (2007), Sawyer (2014)]. 
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In this research I will, from a new perspective, analyse meaning construction as the most significant 
production of learning based on constructivism. 1 will analyse the semantic loop that the learner (and mentor) 
move through in order to construct their personal meanings. I will deal with how the learner (mentor) (i) 
organises the personally constructed concepts, (ii) produces meaningful meanings and (iii) attains a deeper 
comprehension. I will finally provide a framework in order to demonstrate different steps of meaning 
construction based on personal constructed concepts within constructivist interactions. 


2. CONCEPT 

I emphasise that there is a general problem concerning the notion of ‘concept’, in linguistics, in psychology, 
in philosophy, in metaphysics, in computer and information sciences, but, for now, I assume the use of the 
expression concept to be comprehensible in the context. Walter Parker 1 writes: "concepts are the furniture of 
our minds. A well furnished mind is a source of success and lifelong learning. When a student forms a 
concept from its examples, he or she knows more than the definition of a term. This is deep conceptual 
learning rather than superficial knowledge of a vocabulary word.” He also says that "a concept is defined by 
critical characteristics shared by all examples of the concept. For something to be an example of a concept, it 
must contain all these critical characteristics” [Parker (2008)]. 

Generally a concept is an unifying theme for something. In ontology a concept is a fundamental category 
of existence. Following [Margolis and Lawrence (2011)], concepts could be understood as the mental 
representations, where concepts are entities that exist in the brain. They could also be understood as abstract 
objects, where objects are the constituents of propositions that mediate between thought, language, and 
referents. In my research, a concept is an idea which corresponds to some ’distinct entity’ or ’class of entities’ 
or to its ’essential features and attributes’. It can determine the application of a term (especially a predicate), 
and thus plays a part in the use of reason or language [cf. Rudolph (201 1), Baader et al. (2003)]. Analytically 
a concept as a linkage between linguistic expressions and the mental images (representations of the world, of 
inner experiences etc.) that humans have in their minds [Gotzsche (2013)]. I focus on concepts (classes) 
because concepts and the relationships between them are used to establish the basic terminology adopted in 
my modelled pedagogical domain regarding the hierarchical structure. For example, relying on Description 
Logics [Baader et al. (2003)], the concept ‘Mentor’ can be analysed as a concept description (descriptions 
mainly follow the inductive rules) that demonstrates the mentor as a person who has a learner and the learner 
is a person. Formally: Mentor = Person n BhasLearner.Person . This concept construction is able to support 
the formal, explicit specification of a shared conceptualisation of Mentor based on the constructor’s 
conceptions. 


3. DEFINITION & DEFINITENESS 

A definition could semantically be seen as a kind of equation whose left-hand side is a concept and whose 
right-hand side is a description. They are used to introduce symbolic names for complex descriptions. In a 
pedagogical system learners and mentors could define something based upon multiple concept descriptions. 
Actually, these definitions could be constructed based on their own conceptions and background knowings. 
Logically, a set of definitions is (and must be) ’explicit’ and ’unequivocal’, ie. not vague and not ambiguous. 
In fact, ’explicitness’ and ’unequivocality’ are the prerequisites and preconditions for definiteness. So, 
actually, a set of someone’s definitions in a category of her/his constructed concepts could provide a 
backbone for any construction. Then the provided backbone supports the person in defining more complex 
concepts and descriptions over abstract concepts. Subsequently, the learner (mentor) could employ inductive 
rules on her/his personal definitions of abstract concepts in order to produce more complex definitions for 
more complicated described concepts. For example, considering the description ’ MaleMentor = Person n 
Male n 3hasLearner.Person\ the ‘ MaleMentor has been defined by being associated to a description. 


1 http://teachinghistory.org/teaching-materials/teaching-guides/25 1 84 
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4. SEMANTIC INTERPRETATION 

Generally the act of elucidation, explication and explaining the meaning of something is called interpretation 
[Simpson and Weiner (1989)]. Humans need to attempt to provide a way to determine the truth values of 
sentences. Linguistically, interpretation is the continually adjusted relation between the conventional 
meanings of sentences/statements and the actual mental universe of the individual (based on accumulated 
experience of that individual). Logically, an interpretation is an assignment of meanings to non-logical 
symbols. For instance, it can not assign meaning to logical symbols [Not (“0 , And (n), Or (u). Equality (=), 
Equivalency (=)}. Actually we can not assign any meaning to a description until the non logical symbols are 
given interpretations. Considering C and D as two concepts and R(C,D) as any possible binary relationship 
between them, one could have different types of interpretations based on them. More specifically, the 
interpretation / assigns to C a set that contains the interpretation(s) of C, and it also assigns to R a binary 
relation between the elements of two sets (interpretation(s) of C and interpretation(s) of D). Actually, in 
translating from an informal (commonly English) language into a formal language, we need to provide 
symbolisation keys, which are the interpretations of all the non-logical symbols we use in the translation, see 
[Prior (1955)]. In case a given interpretation could assign the value TRUE to a sentence (or theory) that 
interpretation is called a model of that sentence (or theory). In fact, designing a proper model can make the 
definitions adequate. 


5. INTERACTION & MEANING CONSTRUCTION 

A person who undertakes to learn (train) something within a constructivist interaction primarily focuses on 
’concept formation’. S(he) initiates to ask some questions and answer other questions asked by the 
interlocutor. In my research, the most significant necessity of modelling meaning construction in a 
constructivist interaction is to consider the personal mental structures (schemata). They have been created 
regarding what the participants (both learner and mentor) have been affected by. The person inductively 
develops her/his mental entities. The learner (mentor) finds new concepts for herself/himself. Hearing 
different words from the interlocutor could be conducive to new conceptions. A person may have formed a 
concept before participating in the constructivist interaction and then, regarding the feedbacks produced by 
interlocutor, tackles to reform them. So, in fact, ’forming’, ’transforming’ and ‘reforming’ concepts are three 
significant matters in constructivist interactions. The individual has to ‘generalise’ from different examples 
and this may lead her/him to discovering new concept(s). S(he) searches for and lists attributes and properties 
that can be used to distinguish exemplars from non exemplars of various concepts (classes). But what s(he) 
really does is more than generalising from different examples that s(he) hears or produces. More specifically, 
s(he) identifies, specifies and relates the generalised examples and ‘compares’ different examples. In fact, a 
very efficient way to form a new concept and induce new categorisation rules in constructivism is to compare 
a few individuals when their categorical relation(s) is known. On the other hand, s(he) could be able to make 
her/his personal labels of categorising the concepts in order to direct and employ different classes of 
concepts. 

5.1 The Semantic Process 

At the beginning the person proposes some ’schemata’ as different types of concrete actions. Then s(he) 
gradually develops and divides them into more abstract concepts (conceptual entities). A proposed schema 
describes a pattern of the person’s thought. 1 have already said that s(he) could either categorise different 
concepts or follow the categorised concepts. So, in fact, her/his proposed schema could support her/him in 
managing those concepts. I label the first semantic phase Concept Construction (presented in the Figure 1), 
where the individual constructs her/his personal concepts and conceptions based on her/his personal 
schemata. Subsequently, s(he) needs to focus on the ’reflection’ of prior knowledge (what have been 
acquired/created before interacting) and new knowledge (what is being acquired/created during the 
interaction) and the initial meanings. So, in fact, s(he) ‘searches for the (initiative) meanings’ of the class/ 
classes of constructed concepts and their significant relationships. See ‘Search for Meaning’ in the Figure 1. 
So s(he) defines her/his constructed concepts and searches for the initial definitions for the constructed 
concepts. Consequently, s(he) focuses on the ’interpretation’ of the initial meanings and definitions. From the 
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logical point of view, the interpretation of a constructed concept is a ’function’. Generally this function 
assigns a ’meaning’ to a ’symbol’. Formally: Interpretation: Meanings Symbol. The Interpretation 
functions operate the person’s definitions based on her/his constructed concepts. Therefore, they ’activate’ 
the meanings. This phase is presented as Semantic Interpretation in the Figure 1. Accordingly, concerning 
‘Interpretation: Meaning — » Symbol meaning is the product of the inverse of interpretation function 
{Symbol — > Meaning). I label this phase Meaning Balancing (see the Figure 1). There is a strong relationship 
between Semantic Interpretation and Meaning Balancing. The person could be able to balance and adjust the 
initial meanings based on the interrelationships between ‘interpretation’ and ‘inverse of the interpretation’. 
The conclusions make an appropriate background for verifying the personally found meanings based on 
personal constructed concepts. Meaning Balancing is quite supportive in balancing the personal definitions 
and vice versa, see ‘Meaning Formulation’ in the Figure 1. The person formulates the balanced meanings 
based on the balanced definitions for her/his personal constructed concepts. There is an appropriate 
relationship between formulated meanings and balanced definitions. In fact, a meaning would be given a 
better shape after checking the balanced definitions. The formulated meanings organise and reinforce the 
mental structures that the learner (mentor) uses them, as the pattern of her/his thought, in order to develop the 
individual conceptual knowledge. Subsequently, the formulated meanings are some applicable prerequisites 
for Meaningful Conceptual Structuring (see Figure 1) upon personally formulated meanings based on 
personally constructed concepts. On the other hand, the meaningful conceptual structures could induce new 
formulated meanings on higher conceptual levels (presented by the dashed arrow in the Figure 1). And, 
furthermore, the new formulated meanings are considered as new schemata in constructing higher levels of 
conceptions. Finally, the meaningful conceptual structures support her/him in providing meaningful 
meanings. The meaningful meanings highly reflect on the constructor and support her/him in proposing the 
modified schemata on higher conceptual levels. So, the person moves through this semantic loop in order to 
organise her/his personal constructed concepts and construct her/his personal meanings and produce 
meaningful meanings. 



Figure 1. Meaning Construction Within Constructivist Interactions 


6. CONCLUSIONS AND FUTURE WORK 

Constructivism has been known as a learning philosophy and/or a model of knowing. Concerning 
constructivism, two persons whose insights are based on their preconceptions and on personal knowings can 
actively participate in an interaction. The most important objective is to construct their personal knowledge, 
to learn from each other and to train each other. Therefore, they have an opportunity to attain a deeper 
personal understanding, comprehension and greater motivation. In this research I have focused on ‘meaning 
construction’ as the most significant production of learning based on constructivism. I have worked on a 
semantic loop that the intentional participant in a constructivist interaction moves through. S(he) constructs 
her/his personal concepts based on concept formation, defines them (produces individual definitions based on 
constructed concepts), and organises concepts and definitions in order to construct meanings and produce 
meaningful meanings. Meaningful meanings support her/him in constructing knowledge, producing 
meaningful comprehension and reacting more appropriately in front of the interlocutor’s acts. 1 have 
provided a framework for analysing meaning construction based on individual comprehension and personal 
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concept constructions within constructivist interactions. In fact, the proposed loop semantically transforms 
multiple constructed concepts into meaningful meanings (and meaningful comprehensions). It could be 
observed as a new scheme for interpretation based on semantics and on interpretation. Obviously, the 
proposed semantic loop is self-organised. Equivalently, it promotes itself on higher conceptual levels. 

In future research I will focus on the logical analysis of meaning construction within constructivist 
interactions and work on its formal semantics. I will employ some fundamental descriptions in Concept 
Language (Description Logics : DLs) in order to analyse multiple semantic concepts within my progress and 
provide a DLs based formal semantics for analysing meaning construction in constructivist interactions. 
Subsequently, I will be concerned with semantically analysing meaning construction based on personal 
knowings and personal concept constructions in constructivist interactions. I will check the validity of the 
logical descriptions in conceptualising constructivism concerning the ’Structure of Observed Learning 
Outcomes (SOLO)’. The consequences will make a backbone for better conceptualisation of human’s 
understanding. And, the results will be employed in the analysis of formal semantics in terminological 
knowledge for pedagogical knowledge representation systems. They can conceptually analyse pedagogical 
developments in the framework of constructivism and in the context of interactions for promoting human’s 
understanding. 
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ABSTRACT 

This paper reflects on the systematic literature review paper (in progress), which analyzes technology enhanced 
collaborative and cooperative learning in elementary education worldwide from 2004 to 2015, focusing on the 
exploration of technology mediated group formation. The review paper reports on only few cases of technology 
supported methods of group formation, since in the majority of analyzed papers groups are mostly formed before the 
activity by teachers or researchers. This paper suggests a novel approach by proposing an adaptive model for technology 
supported group formation. 


KEYWORDS 
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1. INTRODUCTION 

Collaborative learning is "a situation in which two or more people learn or attempt to learn something 
together" (Dillenbourg 1999), whereas in cooperation partners split the work, solve sub-tasks individually 
and then assemble the partial results into final output (Johnson & Johnson 1999). In order to work together 
members can be grouped based on various parameters such as gender, learning style or aptitudes forming 
homogenous or heterogeneous groups. In the context of time groups can be periodically active (for example 
every week) or continuous through the whole period of the activity. Furthermore, group members can even 
change during the activity. 

This paper focuses on the exploration of technology supported methods of group formation before or 
during the collaborative or cooperative activity, focusing on elementary education worldwide, analyzing 
results obtained from a literature review research paper (in progress). The review paper resulted in 118 
articles addressing technology supported collaborative and cooperative learning in elementary education of 
which 5 report on system mediated group formation. 


2. GROUP FORMATION MODEL 

Based on the review results, the authors detected and defined four methods of group formation which is 
either organized before the activity or mediated during the activity by the system: 1) system mediated 
heterogeneous pre-grouping (Huang et al. 2011) where learners are assigned to groups by the system based 
on their responses to a learning styles questionnaire given before learning activities, 2) random pre-grouping 
by the system before the activity (Wyeld et al. 2012), 3) dynamic grouping (Nussbaum et al. 2009) as defined 
by Zurita (Zurita et al. 2005) where members are grouped “during collaborative activity for reaching a given 
educational objective”, and 4) a variant of dynamic grouping (Wong et al. 2011; Boticki et al. 2013) where 
groups are dynamically established prior to the learning activity. Selected papers are presented in the Table 1. 
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Table 1. Research papers with automatized group formation and the accompanying grouping characteristics. 


Article 


(Nussbaum et al. 2009) 
(Wong et al. 2011) 
(Huang et al. 2011) 
(Wyeld et al. 2012) 
(Boticki et al. 2013) 


Time of group formation 


During the activity 

Pre-organised 

Pre-organised 

Pre-organised 

Pre-organised 


Group characteristics Group formation 


Heterogeneous Dynamic and random 
Heterogeneous Random 

Heterogeneous Random 

Heterogeneous Random 

Heterogeneous Random 


The authors of the paper propose an adaptive model for dynamic group formation (Figure 1). In order to 
form the groups, the system collects the data about individual learners using this model. The data is analyzed 
and processed with an adequate learning analytics algorithms and adaptation module to be later used in 
forming adequate groups for collaborative or cooperative tasks. Once the activity starts, the system collects 
both individual and group data and dynamically regroups peers if needed. Regrouping outcome can be the 
termination of group work in order to start individual work assignments, merging of two or more different 
groups or the combination of group members with members of another group. 
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Figure 1. The adaptive grouping mode 
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ABSTRACT 

Case study instruction is characterized by centering instruction around the use of hypothetical classroom dilemmas. It 
uses descriptive stories and invites students to discuss application of course material as they engage in hypothetical 
classroom problem-solving and teacher decision-making. Teaching is a complex profession that requires high levels of 
multitasking in order to successfully instmct students and handle classroom disruptions. Because most pre-service 
teachers do not possess extensive classroom experience when enrolled in teacher education, working with case studies 
provides opportunities to test theoretical concepts and speculate on making decisions that would affect their future 
classrooms. The purpose of this paper is to discuss the elements of case study instruction that we have found to be 
effective in encouraging online learner’s self-regulated learning for educational psychology. 


KEYWORDS 

Case instruction, online learners, educational psychology 


1. INTRODUCTION 

Case study learning can be viewed as a constructive approach to teaching in which learning, from a 
sociocultural perspective, takes place in context. Because cases can be written from various perspectives, 
they offer instructors a multitude of real-world scenarios from which students can learn from while also 
demonstrating the collaborative nature of teaching by illustrating how a teacher works jointly with the school 
principal, guidance counselor, and students’ parents in order to assist students (Ching, 2011). The use of case 
instruction in educational psychology, in particular, is a “natural fit” with constructivist principles that views 
learning as “interactive, socially constructed, collaborative, and problem-related (Sudzina, 1997, p. 207). In 
addition, case-study instruction has many benefits including student learning, problem-solving, 
decision-making, critical thinking, self-efficacy for educational situations, cognitive flexibility, and 
perspective-taking as well as enlightening them to the complex nature of teaching (PytilikZillig, et al., 2011; 
Ching, 2011; Sudzina, 1997). The published literature discusses a number of ingredients for effective case 
instruction: collaborative learning, case complexity, perspective-taking, scaffolding critical thinking, 
revisiting cases and authentic assessment. 


2. CASE INSTRUCTION AND ONLINE LEARNING 

According to the National Center for Education Statistics (NCES), from 2000 to 2008, the percentage of 
undergraduate students enrolled taking at least one distance education course increased 12% (from 8 percent 
to 20 percent) (Radford, 2011). This ascending curve in enrollment is projected to continue, thereby 
presenting the education community with numerous challenges (Allen & Seaman, 2008). Case discussion 
can be adapted to the online environment and could take place in synchronous or asynchronous discussion 
boards and chat-rooms. Past research also indicates case studies can have positive results for educational 
psychology students’ involvement, communication, peer support, and reciprocate interaction among 
stakeholders that includes students, instructors, and cooperating teachers (Bonk, et al., 1998). In fact. 
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students engaged in either traditional or online case discussion improved in their ability to critically analyze 
novel cases and outperformed students not being taught using case studies ( PytlikZillig, et al, 2011). One 
element of case instruction particularly important to encouraging online learner’s self-regulated learning for 
educational psychology is case complexity (Gonzalez-DeHass, Willems, & Vasquez-Colina, 2014). Case 
complexity refers to the case’s intricacy and ability to engage pre-professional teachers in problem-solving 
for real-life situations. It is possible that when students see more utility in what they are learning, they are 
more willing to apply themselves and take an active interest in their learning. This finding mirrors what 
others have found in that cases can be intrinsically motivating and promote self-regulated learning if they are 
realistic, personally relevant, and where activities are seen as valuable and interesting ( DeMarco, Hayward. 
& Lynch, 2002). 


3. POTENTIAL INSTRUCTIONAL IMPLICATIONS 

The purpose for this proposal is to take the elements of case study instruction that we have found to be 
effective encouraging online learner’s self-regulated learning for educational psychology and investigate 
teaching strategies that can be used to further scaffold and encourage students to work collaboratively with 
cases in the online environment. This will include discussion of video-based cases to test theoretical 
knowledge and encouraging students to participate in collaborative discussions. Computer programs allow 
for video cases to be manipulated by the instructor to add questions at various points in the case, allowing for 
the instructor to test students’ individually or to offer the opportunity for whole-group collaborative 
discussions in real-time. Instructors could also design sessions in which only small-groups of students 
discuss a particular case while he or she scaffolds their learning. Or cases could be used as part of a 
culminating written project where students reflect on their learning as they complete independent authentic 
case analyses. Using cases in this manner is directly linked to case complexity, offering instructors various 
ways in which to expose students to real-life teaching challenges, the multiple perspectives of the different 
stakeholders involved in the case, and the applicability of diverse theoretical perspectives in educational 
psychology. These potential strategies add to the discussion of how instructors can incorporate case 
instruction in the online environment most effectively. 
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ABSTRACT 

Online education has a relatively short history in the grand history of education. With options for online delivery modes 
developed over the last two decades, understanding student motivations for choosing one option over another will be 
helpful to any institution of higher education planning new offerings. This reflective paper presents the beginning aspects 
of a study at one university in the United States, using a prior study from Australia as a guide. 

KEYWORDS 
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1. INTRODUCTION 

While distance learning has a long history, offering courses through online options goes back to when the 
Internet was effectively available through Usenets and the like. Only since 1993 with the advent of the World 
Wide Web that both synchronous and asynchronous interactions between students and between students and 
instructors became available (Ifenthaler, Bellin-Mularski, & Mah, 2015). Over the last 20 years, studies in the 
United States have mostly focused on how higher education faculty perceived either teaching through online 
or blended options or the reputation of online courses in general (Allen & Seaman, 2013, 2006; Schifter, 
2005. And while there is a wealth of research on why students choose their institution of higher education 
(Harkera, Sladea, & Harkera, 2001; Cebula & Lopes, 1982; Chapman, 1981; Booth, 1997), little research has 
been concerned with what motivates students to study through different modes of delivery, including 
face-to-face, fully online, blended between online and face-to-face (Bailey, Ifenthaler, Gosper, Kretzschmar, 
& Ware, 2015). 

In a paper presented at CELDA 2014, Bailey and colleagues (2015) presented a study of Australian 
institutions of higher education and motivation for students to study in different modes of delivery. The 
survey asked students taking courses through different modes of study about motivation to take courses 
through online modes. This study attempts to replicate the same study, but at one U.S. institution. Temple 
University in Philadelphia. 

The research questions for this study are, what factors influence a Temple University student to take a 
course through online modes of delivery/learning? And. what technologies are used currently, and which 
would these students like to see used within a course delivered by online modes? Ultimately, outcomes 
would facilitate sound programmatic planning that accounted for reported student circumstance as well as 
their technological preferences. 


2. METHODS 

The survey used by Bailey and colleagues (2015) was adapted and used with Temple University online 
students. The survey was reviewed by the Office of Assessment, a pilot study was completed in August 2015 
to clarify items, and the full survey was implemented in last September 2015 to all students identified as 
taking online courses at Temple University. This poster will present the initial findings of that study. 
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3. DISCUSSION 

While the results of this study are incomplete, we anticipate the opportunity to compare findings at one U.S. 
institution with that from the Bailey, et al. (2015) study from Australia. This will be the first international 
comparison of factors influencing students choosing different modes of study. In addition, the findings will 
be useful by the University in planning and developing future programs to be delivered through online 
modes. 
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ABSTRACT 

The undergraduate Business and Eluman Development (HD)Departments at California State University San Marcos 
(CSUSM), are collaborating in an innovative interdisciplinary approach to supporting internships with local businesses in 
order to provide college seniors with experiences in the area of career development known as the “Senior Experience.” 
Three local companies associated with serving the community in the area of healthcare or counseling, will hire a team of 
5 college seniors (3 Business majors and 2 E1D majors) to create a project for the company. The selected project will be 
agreed upon and supervised by both the company and university representatives from both departments. This paper will 
reflect on the literature base pertaining to creating a successful collaborative interdisciplinary internship for college 
seniors exploring possible career options. 

KEYWORDS 

Collaborative Internships, Career Development 


1. INTRODUCTION 

As study after study emphasizes the importance of supporting learning outside the confines of a classroom, 
one viable method is establishing internships. Internships in local companies provide students opportunities 
with actual experience within an organization, making students more employable. 

The undergraduate Business Administration and Human Development (HD) departments at California 
State University, San Marcos (CSUSM) are collaborating in an innovative interdisciplinary approach to 
creating internships known as the “Senior Experience.” Three local companies will each pay $1500 to 
CSUSM to hire a team of 5 college seniors to create a project for the company. These projects will be agreed 
upon by the company and the university. These rigorous and real world projects provide the opportunity for 
college seniors to act as consultants by actually applying the culmination of classroom. In addition, the 
collaborating businesses will gain with focused college seniors who provide a fresh perspective on the 
selected project. The three different projects will be in the area of “health” or “counseling”. At the end of the 
semester, all teams participating in the Senior Experience will present their projects via a power point 
presentation. In addition, students will participate in a university hosted “Trade Show” to share their 
experiences and accomplishments with their assigned businesses. This reflection paper will focus on the 
development of this project. 


2. BODY OF PAPER 

Internships 

There has been an increase in the number of students participating in internships from 3% of all college 
students in 1980 to 75% in 2012 (Hurst, et al. 2012). An internship is defined by the National Association of 
Colleges and Employers (NACE, 2015) to include the following: 1.) To be an extension of the classroom 
learning; 2.) To provide skills or knowledge to be applied to other employment settings; 3.) A specific 
beginning and end, with a job description and desired qualifications; 4.) The student has identified specific 
learning goals and objectives; 5.) Supervision by an individual trained and experienced in the field of 
endeavor is provided to the intern; 6.) Routine feedback is provided by the supervisor directly to the intern; 
7. ) Resources, equipment and facilities provided by the employer support the learning goals and objectives of 
the intern. In addition, academic internships involve the student, employer and the university (Daugherty, 
2000). The work completed with the company must be related to the student’s major and meet the provided 
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criteria for hours worked, duties performed and the relation of these activities to studies, in order to receive 
any academic credit. These valuable internships provide opportunities for experiences in both the academic 
and the professional setting for college students. Providing services in a selected career area provides a direct 
understanding of what this particular career would be like on a day to day basis. For college seniors, this 
could be a life changing experience! 

Partnerships 

In order to support students participating in an internship, multiple partnerships are required between the 
university, the placement sites and the students. Frequent and effective communication between all partners 
involved is essential to support student interns in successfully completing the assigned project. 

Another important aspect of this partnership lies within the area of mentoring. According to Johnson 
(2002), mentors influence the professional development of interns through teaching, advising and/or 
supervising both learning and performance outcomes. For the CSUSM Senior Experience, two faculty 
members and the site supervisor will all help mentor the students. 

Feedback 

Feedback will be provided to the interns by both the local business representative and the team’s 
university supervisor. When considering the important criteria that need to be established and included in the 
feedback to interns, NACE (2015) found that business supervisors for interns wanted more clarity with the 
university expectations and more effective guidance on evaluating interns; more specific focus on creating an 
effective match between interns and their internship; and for the university to offer more career training 
experiences to support the development of professional behaviors in interns. These suggestions will be 
considered for guiding feedback from the business supervisors to the interns involved with the CSUSM 
project. 

Promoting Intern Success and Beyond 

A study conducted on job significance for interns indicated that the significance of specific task 
completion required by the intern, as well as the feedback provided on how well the task was completed, was 
indicated as being most significant for interns (D’ Abate, Youndt & Wenzel, 2009). The CSUSM project will 
support students in approved goal and objective completion which will be mentored by both faculty. 

Other aspects of the work environment that were found to be significant in the experiences of interns 
consisted of learning opportunities, supervisor support and organization satisfaction (D’Abate, Youndt & 
Wenzel, 2009). In addition to these considerations of student needs, collaborating with a selected team of 
students also provides an effective route for the company to identify prospective employees. 


3. CONCLUSION 

According to NACE (2015), college graduates with internship experience are much more likely to receive a 
job offer (approximately 57% in 2013), as well as a higher starting salary than students without internship 
experience. To turn an internship into a job, Adams (2015) recommends that a student select an internship 
that requires substantial work; act professionally; network; ask questions; set goals; volunteer; and follow up 
with the business after the internship is completed. Using previously conducted empirical studies as a basis, it 
is the goal of the CSUSM Collaborative Senior Project to prepare, support and assist university seniors in 
successfully attaining the career of their dreams. 
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ABSTRACT 

This reflection paper argues that the design and development of digital games teach essential 21 st century skills. Intrinsic 
to application and game development is design thinking. Design thinking requires iterative development, which demands 
creativity, critical thinking and problem solving. Students are engaged through learning by doing in both concrete and at 
the same time in abstract thinking. Application and game design requires systems thinking that gives context and 
meaning to narrative and story telling. A deep understanding of information, media ( visual, auditory and performative) 
and technology coupled to mathematical skills is a prerequisite to success in this domain. Application and game 
development requires social intelligence and interpersonal skills for working in diverse teams where communication and 
collaboration is essential. Finally game design requires knowing oneself as a designer but also knowing and 
understanding perception, psychology and cognition of the user and the player. 
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1. INTRODUCTION 

The game design document is similar to an extensively researched term paper. It is an attempt to describe a 
proposed game, in its entirety and in detail before any work begins. Ken Wong, the lead designer for the 
iPhone game Monument Valley explains why game documents are not useful: “We find you need to make a 
game wrong at least two or three times before you find the right path. We took a lot of opportunity to design 
and explore, knowing that a lot of it would be thrown away.” The development of digital games is an 
iterative process that requires creativity, exploration of promising avenues, dead ends and a willingness to 
fail in order to make progress. 


2. DESIGN THINKING 

Design thinking is problem solving conditioned by learning by doing. Design begins with brainstorming and 
idea generation using free association where any idea is fair game. A second pass selects a subset of likely 
candidates. Ideas are captured as sketches and notes on paper or whiteboards making it possible to see where 
changes can be made and how variations can be generated by similarity, contrast or even systematic 
combinatorics. Iteration is a process of generation, trial and error. Errors point the way to success. 

2.1 Digital Game Design Thinking 

Games begin first with brainstorming out of which emerges a core concept often coupled to an activity or 
interaction. Once implemented in code this coupling becomes a game mechanic. However for every game 
mechanic the game designer must test and iterate to find the right balance between engagement, difficulty, 
ease of use and boredom while advancing both the storyline and the gameplay. 
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2.2 Learning by Doing 

It is not enough to think of a game mechanic, it must be built so it can be tested. Building it requires 
analytical, mathematical and concrete thinking needed for programing. For example a character may have the 
property of health. Concrete thinking is required to determine the data type (floating point or integer) for data 
representation. There is a leap of abstraction with the creation of a variable to store the value. In a further 
abstraction a function can pass this variable to increment it or decrement it. This abstraction is taken a step 
further when it represents the “health” of a character in the game. As increasing levels of abstraction and 
representation are added in a game where the game becomes a simulation of a complex world with physics, 
autonomous characters and programed entities interacting with the player. 

2.3 Game Design is a Team Sport 

Game design and development requires game designers, programmers, writers, artists, sound designers, 
composers, content experts, human factor experts, cognitive psychologists, marketing specialists and legal 
experts. It is by definition a collaborative process that requires clarity of written and verbal communication. 
Effective teams require members that have highly developed social intelligence and interpersonal skills 
which fosters understanding of teammates and the audience of players of diverse backgrounds and gender. 

2.4 Know Thyself 

Game Design teaches essential life and career skills. Team members must set goals, manage workload while 
working collaboratively yet independently. Self-assessment, an open mind, curiosity and self-directed life 
long learning are essential. With career advancement comes the responsibility to mentor and lead others. 


3. CONCLUSION 

Learning Game Design prepares students with 21 st Century innovation skills and abilities for a rapidly 
changing technological world. These include critical thinking and problem solving; systems thinking; 
learning how to make judgments and decisions. Creativity and innovative thinking is fostered by idea 
generation techniques as well as cultivating curiosity and openness to different perspectives. Team work 
means working effectively, collaboratively and creatively with others; having clear verbal and written 
communication skills; being respectful of difference in the workplace and welcoming of a diversity of 
gender, lifestyle and cultural backgrounds. Information, Media and Technology skills are of course 
prerequisite to the field; life and career skills (flexibility and adaptability; time management, self-directed 
learning) combined with accountability sets the stage for greater responsibility and success. 
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ABSTRACT 

In this research the author defines critical thinking as skills and dispositions which enable one to solve problems logically 
and to attempt to reflect autonomously by means of meta-cognitive activities on one's own problem-solving processes. 
The author focuses on providing meta-cognitive knowledge to help with self-assessment. To develop students’ critical 
thinking, it is important for students to be able to use critical thinking rubric and assess themselves. Critical thinking 
self-assessment system has three components, video recording system using a wearable device, critical thinking rubric 
and meta-cognitive activities. 
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1. INTRODUCTION 

To develop students’ ability to work together to solve problems is an important factor in education. The 
ATCS21 Project proposed 21st century Skills as ways of thinking, tools for working, ways of working and 
ways of living in the world (ATCS21 ). Recently, research into the assessment and teaching of 21st century 
skills has forged ahead, and MOOC learning environments have also been provided (e.g. at the University of 
Melbourne). According to ATCS21, critical thinking is one aspect of ways of thinking. A number of 
researchers such as Dewey (1910), Glaser (1941), and Ennis (1985) define critical thinking as reflective and 
logical thinking. The Association of American Colleges and Universities (AACU) defines critical thinking 
as a habit of mind characterized by the comprehensive exploration of issues, ideas, artifacts, and events 
before accepting or formulating an opinion or conclusion. AACU also provides a rubric known as a value 
rubric as a critical thinking assessment tool. The critical thinking value rubric has five categories: 
explanation of issues, evidence (selecting and using information to investigate a point of view or conclusion), 
influence of context and assumption , student's position (perspective, thesis/hypothesis) and conclusions and 
related outcomes (implications and consequences). The value rubric is used to rank students' activities on four 
different levels. For instance, the highest level (capstone) in the evidence category is ‘Information is taken 
from source(s) with enough interpretation/evaluation to develop a comprehensive analysis or synthesis. 
Viewpoints of experts are questioned thoroughly’. On the contrary, the lowest level (benchmark) is 
‘Information is taken from source(s) without any interpretation/evaluation. Viewpoints of experts are taken as 
fact, without question. ’To develop their critical thinking, it is important for students to be able to use this 
rubric and assess themselves. 

The author focuses on providing meta-cognitive knowledge to help with self-assessment. In this research 
the author defines critical thinking as skills and dispositions which enable one to solve problems logically 
and to attempt to reflect autonomously by means of meta-cognitive activities on one's own problem-solving 
processes. To provide information on meta-cognitive activities, a record of the learning process is required. 

One approach for recording this information is using computer log data. Log data about students’ 
activities such as browsing the web, writing comments, reading web pages and discussing with peers on the 
bulletin board system will provide useful information. These data have already been used with assessments 
such as PISA and the National Assessment of Information Utilization Ability in Japan (2015). 
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On the other hand, computing is merely one aspect of students’ activities. Students also read books, write 
comments or memos, and hold discussions with peers and teachers. Furthermore, computer log data are of no 
direct use in students' meta-cognitive activities. To provide information needed for meta-cognitive activities, 
the entire record of the learning process is required. Students’ video-recorded data would seem to be 
particularly useful in assisting their meta-cognitive activities. In addition, the author mentions individual 
differences among students. It has recently been discovered that there are differences among individuals with 
regard to critical thinking (Gotoh . Individual differences such as personality, disposition toward critical 
thinking and self-efficacy will be discussed). 


2. METHOD 

Elementary school children, junior high school students, senior high school students and university students 
took part in this study. The material selected was cooperative problem solving, such as integrated studies and 
the knowledge constructive jigsaw method. Elementary school children at an elementary school attached to 
Niigata University gathered information about how to promote Niigata City and proposed a promotion plan. 
Niigata University students took part in the knowledge constructive jigsaw method and discussed their own 
career design from the point of view of technical / functional competence, general managerial competence, 
autonomy / independence, security / stability, entrepreneurial creativity, service / dedication to a cause and 
pure challenge. 

2.1 Learning Environment 

This system has three components of which the first is a video recording system using a wearable device. To 
record the entire learning process, a wearable device such as Google Glass or Sony Wearable Glass is used. 
A lesson protocol was developed using the recorded video. Using these data, the researcher extracted samples 
that meet the requirements of the rubric described below. 

2.2 Critical Thinking Rubric 

The second component is the critical thinking rubric. As already mentioned, previous research exists on the 
critical thinking rubric (Griffin, et.al 2012, Beyer 1985, Miyake 2014, VALUE project). The critical thinking 
rubric was used as a tool to enable students to carry out self-assessment and reflection. It is important for 
students to be able fully to understand and use the critical thinking rubric. For this reason, the critical 
thinking rubric was modified to suit each school level. For instance, the university rubric has four levels: 
capstone, upper milestone, lower milestone and benchmark similar to the VALUE Rubric. In contrast some 
elementary school rubrics have only two levels. 

2.3 Meta-cognitive Activities 

The third component consists of meta-cognitive activities. The integrated study project consists of several 
small units. At the end of each small unit the author requested students to reflect on their learning from the 
point of view of their own critical thinking performance. To assist these activities, students’ watch a video 
that meets the requirements of the critical thinking rubric. The video of the sample extracted by the 
researcher undoubtedly helps students with their self-assessment. After watching the video, students assessed 
their own learning using the critical thinking rubric. At the university level, samples were displayed in which 
information is taken from source(s) with enough interpretation/evaluation to develop a comprehensive 
analysis or synthesis and viewpoints of experts are questioned thoroughly. Students and teachers discussed 
points in the samples of students' activities where students showed excellence. Discussion provides a 
viewpoint from which students are able to assess their own learning activities. Students also become aware of 
their strengths and weaknesses. 
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3. EXPECTED RESULT 

A pilot study was started at Niigata University and at the elementary school attached to Niigata University. 
At the elementary level. It was found that some children felt that self-assessment without any video 
recording was difficult. A wearable device seems to be useful because these devices provide a natural angle 
and clear sound. At the university level, these devices provide supportive information to evaluate students’ 
activities. In terms of evidence, one of the critical thinking categories, the upper milestone was: ‘Information 
is taken from source(s) with enough interpretation/evaluation to develop a coherent analysis or synthesis and 
viewpoints of experts are subject to questioning’ whereas the lower milestone was: ‘Information is taken 
from source! s) with some interpretation/evaluation, but not enough to develop a coherent analysis or 
synthesis. Viewpoints of experts are taken as mostly fact, with little questioning’. 

A movie of the learning process recorded by a wearable device will provide useful information to enable 
a more precise understanding of the differences in these levels. Meta-cognitive activities and reflection will 
enable students to attain a deeper level of self-assessment. Differences among individuals are also taken into 
consideration. In particular, the difference in the problem solving process between those who are highly 
disposed towards critical thinking and those who are not, is not yet known. Using a critical thinking 
disposition scale the author extracted two types of students and compared problem- solving process, 
meta-cognitive activities and relfection and self-assessment between these two types. 
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